Man-machine interaction system and method of intelligent refrigerator

Abstract

The invention relates to the technical field of intelligent refrigerators, and particularly discloses a man-machine interaction system and method of an intelligent refrigerator, which utilize data collected by a microphone array to perform preliminary noise reduction processing, and then adaptively further purify user interaction instruction voice signals according to simultaneously collected environmental noise information. The two-stage noise reduction process not only considers the basic physical level noise reduction, but also adds the understanding of the context information, so as to achieve a better voice definition recovery effect. And finally, outputting an operation instruction to control the refrigerator by using a voice recognition technology and a command analysis mode. Therefore, the noise reduction strategy can be dynamically adjusted by analyzing the environmental noise characteristics in real time, so that the noise reduction processing of the user interaction instruction voice signal is more effectively carried out, and the extraction of useful information from the original audio data becomes more efficient and accurate; the man-machine interaction accuracy and the user satisfaction degree of the intelligent refrigerator are improved in a more intelligent mode.

Description

Technical Field

[0001] The present application relates to the technical field of smart refrigerators, and more specifically, to a human-computer interaction system and method for a smart refrigerator. Background Art

[0002] With the development of smart home technology, smart refrigerators, as an important part of the home Internet of Things, are gradually becoming an important bridge connecting users and home life. The human-computer interaction system is one of the key factors to improve the user experience, and speech recognition technology plays a vital role in it. However, in practical applications, the diversity of environmental noise poses a huge challenge to speech recognition. For example, different types of noise in the kitchen, such as range hoods, TV sounds, and conversation sounds, will interfere with the clarity of the voice signal, resulting in an increase in the recognition error rate. Therefore, developing a human-computer interaction system that can effectively cope with complex environmental noise is crucial to improving the user experience of smart refrigerators.

[0003] Traditional noise reduction methods for human-computer interaction systems often rely on fixed algorithm models, such as spectral subtraction and Wiener filtering. Although these methods perform well in specific scenarios, a single noise reduction algorithm cannot effectively cope with all situations, and its effect is greatly reduced when facing ever-changing environmental noise. In addition, traditional noise reduction algorithms for human-computer interaction systems usually work based on preset fixed parameters, and it is difficult to automatically adjust their behavior according to the changes in noise characteristics in actual scenarios. For example, for spectral subtraction, it relies on the estimated noise power spectrum density to subtract noise from the speech signal. When the type or intensity of noise changes, if the parameters cannot be adjusted accordingly, it may cause over-subtraction (speech distortion) or under-subtraction (noise residue), affecting the effectiveness of the human-computer interaction of the smart refrigerator and reducing the user experience.

[0004] Therefore, an optimized human-computer interaction system for a smart refrigerator is desired. Summary of the invention

[0005] The present application provides a human-computer interaction system and method for a smart refrigerator, which can dynamically adjust the noise reduction strategy by real-time analysis of environmental noise characteristics, thereby more effectively performing noise reduction processing on user interaction command voice signals, making it more efficient and accurate to extract useful information from raw audio data, thereby improving the accuracy of human-computer interaction and user satisfaction of smart refrigerators in a more intelligent way.

[0006] In a first aspect, a human-computer interaction system for a smart refrigerator is provided, comprising: a user interaction command voice acquisition module, which is used to acquire the user interaction command voice signal using a microphone array built into the refrigerator after receiving a voice interaction start action.

[0007] The command voice signal primary noise reduction module is used to perform primary noise reduction on the user interaction command voice signal to obtain a user interaction command voice signal after primary noise reduction.

[0008] The environmental noise signal acquisition module is used to acquire environmental noise signals using the microphone array.

[0009] The command voice signal adaptive noise reduction module is used to adaptively reduce the noise of the user interaction command voice signal after the primary noise reduction based on the environmental noise signal to obtain the user interaction command voice signal after the secondary noise reduction.

[0010] The interactive command speech recognition module is used to send the user interactive command speech signal after the secondary noise reduction to the speech recognition model to obtain a text description of the user interactive command.

[0011] The command parsing module is used to perform command parsing on the user interaction instruction text description to obtain a command content parsing result.

[0012] The refrigerator operation control feedback module is used to call the operation instruction to control the refrigerator and feedback the operation result based on the command content analysis result.

[0013] In a second aspect, a human-computer interaction method for a smart refrigerator is provided, comprising: after receiving a voice interaction start action, using a built-in microphone array in the refrigerator to collect a user interaction command voice signal; performing primary noise reduction on the user interaction command voice signal to obtain a user interaction command voice signal after primary noise reduction; using the microphone array to collect an environmental noise signal; based on the environmental noise signal, performing adaptive noise reduction on the user interaction command voice signal after primary noise reduction to obtain a user interaction command voice signal after secondary noise reduction; sending the user interaction command voice signal after secondary noise reduction to a voice recognition model to obtain a user interaction command text description; performing command parsing on the user interaction command text description to obtain a command content parsing result; based on the command content parsing result, calling an operation command to control the refrigerator and feeding back the operation result.

[0014] The present application provides a human-computer interaction system and method for a smart refrigerator, which uses the data collected by the microphone array to perform preliminary noise reduction processing, and then adaptively further purifies the user interaction command voice signal based on the environmental noise information collected at the same time. This two-stage noise reduction process not only takes into account the basic physical level denoising, but also incorporates the understanding of contextual information to achieve better voice clarity restoration. Finally, the operation instructions are output using voice recognition technology and command parsing to control the refrigerator. In this way, the noise reduction strategy can be dynamically adjusted by analyzing the environmental noise characteristics in real time, so as to more effectively perform noise reduction processing on the user interaction command voice signal, making it more efficient and accurate to extract useful information from the original audio data, and improving the accuracy and user satisfaction of the human-computer interaction of the smart refrigerator in a more intelligent way. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application are briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0016] Figure 1 This is a schematic block diagram of the human-computer interaction system of the smart refrigerator of an embodiment of the present application.

[0017] Figure 2 This is a schematic diagram of data flow of the human-computer interaction system of the smart refrigerator according to an embodiment of the present application.

[0018] Figure 3 This is a schematic block diagram of an adaptive noise reduction module for command voice signals in a human-computer interaction system of a smart refrigerator according to an embodiment of the present application.

[0019] Figure 4 This is a schematic block diagram of an environmental noise-speech signal feature interaction processing unit in the human-computer interaction system of a smart refrigerator according to an embodiment of the present application.

[0020] Figure 5 This is a schematic flowchart of the human-computer interaction method of the smart refrigerator of an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0022] In view of the above technical problems, in the technical solution of this application, a human-computer interaction system for a smart refrigerator is proposed. Figure 1 and Figure 2 As shown, the human-computer interaction system of the smart refrigerator includes: a user interaction instruction voice acquisition module 10, which is used to collect the user interaction instruction voice signal by using the built-in microphone array of the refrigerator after receiving the voice interaction action; a command voice signal primary noise reduction module 20, which is used to perform primary noise reduction on the user interaction instruction voice signal to obtain the user interaction instruction voice signal after primary noise reduction; an environmental noise signal acquisition module 30, which is used to collect the environmental noise signal by using the microphone array; a command voice signal adaptive noise reduction module 40, which is used to perform adaptive noise reduction on the user interaction instruction voice signal after primary noise reduction based on the environmental noise signal to obtain the user interaction instruction voice signal after secondary noise reduction; an interactive instruction voice recognition module 50, which is used to send the user interaction instruction voice signal after secondary noise reduction to the voice recognition model to obtain a user interaction instruction text description; a command parsing module 60, which is used to perform command parsing on the user interaction instruction text description to obtain a command content parsing result; a refrigerator operation control feedback module 70, which is used to call the operation instruction to control the refrigerator and feedback the operation result based on the command content parsing result.

[0023] Exemplarily, in the user interaction instruction voice acquisition module 10, after receiving the start voice interaction action, the user interaction instruction voice signal is collected using the microphone array built into the refrigerator. It should be understood that by detecting specific start voice interaction actions (such as pressing a voice button or recognizing a wake-up word), the possibility of false triggering can be effectively reduced, ensuring that the system only starts recording when the user really wants to interact with the refrigerator, thereby improving the response accuracy of the system and the user's experience. At the same time, discontinuous monitoring of all sounds in the environment can avoid unnecessary audio data collection and protect the privacy and security of the user. Only when the start command is clearly received will the microphone be activated for recording, reducing interference with the user's private life. Continuous monitoring consumes a lot of computing resources and electricity, while the on-demand activation method can reduce energy consumption while ensuring functionality, extend the service life of the device, and reserve more resources for other tasks.

[0024] In one embodiment, the voice interaction action is initiated by the user pressing a voice button or saying a wake-up word. Specifically, in the present application, a high-quality microphone array is installed at different positions inside the refrigerator to cover the largest possible pickup range and ensure that the user's voice commands can be clearly captured even in a noisy environment. At the same time, a low-power front-end processing module is implemented to monitor in real time whether a preset wake-up word is spoken. Once the wake-up word is detected, the main processor is immediately notified to prepare to receive subsequent instructions. At the same time, in order to optimize the user experience, the system also provides an intuitive feedback mechanism, such as a visual indicator light or a prompt tone, to let users know that their voice has been successfully received and is being processed. In addition, in other examples of the present application, multiple startup methods are also supported. In addition to voice wake-up, physical buttons can also be set as a backup solution to meet the habits of different users. In summary, through the combination of reasonable hardware layout and efficient software algorithms, the user interaction command voice signal acquisition process after receiving the voice interaction action can be effectively implemented, which not only improves the accuracy and efficiency of the interaction, but also enhances user satisfaction.

[0025] Exemplarily, in the command voice signal primary noise reduction module 20, the user interaction command voice signal is subjected to primary noise reduction to obtain a user interaction command voice signal after primary noise reduction. It should be understood that the unprocessed original voice signal may contain a large amount of redundant information, and directly performing complex processing on it (such as deep learning model analysis) not only increases the computing cost, but may also cause delays. Primary noise reduction can filter out unnecessary noise at an early stage and reduce the workload of subsequent advanced processing modules, thereby speeding up the response speed of the entire system. Primary noise reduction provides the basis for the subsequent adaptive noise reduction. It can preliminarily purify the voice signal without relying on specific environmental characteristics, so that the system can work in a more diverse environment, rather than just optimizing for certain fixed scenarios.

[0026] In one embodiment, the instruction voice signal primary noise reduction module is used to: use a frequency domain filter to perform primary noise reduction on the user interaction instruction voice signal to remove the high-frequency noise of the user interaction instruction voice signal to obtain the user interaction instruction voice signal after primary noise reduction. Specifically, the frequency domain filter can more easily identify and remove the noise concentrated in the high-frequency area by converting the voice signal in the time domain to the frequency domain. For example, the voice signal is mapped from the time axis to the frequency axis using a fast Fourier transform (FFT), and then a bandpass or low-pass filter is applied to eliminate the high-frequency part that exceeds the frequency range of human speech. In other examples of the present application, in addition to algorithmic improvements, dedicated hardware accelerators (such as DSP-digital signal processors) can also be used to perform efficient real-time noise reduction operations. This not only improves the processing speed, but also ensures that the noise reduction program can be run smoothly on resource-constrained devices. Although primary noise reduction mainly focuses on fixed noise reduction operations, a certain degree of adaptive mechanism can also be introduced. For example, filter parameters are automatically adjusted based on environmental sound samples within the initial few seconds to better adapt to different usage environments.

[0027] Exemplarily, in the environmental noise signal acquisition module 30, the microphone array is used to collect environmental noise signals. It should be understood that in the human-computer interaction scenario of the smart refrigerator, the home environment is often full of various background noises, such as range hoods, TV sounds, conversation sounds, etc. These noises will interfere with the user's voice commands, resulting in an increase in the error rate of the voice recognition system. Traditional fixed parameter noise reduction methods are difficult to cope with such diverse and dynamically changing noise environments. By collecting environmental noise signals, the system can understand the noise characteristics of the current environment in real time, and adjust the noise reduction strategy accordingly, ensuring that the user's commands can be effectively captured even in a noisy environment, providing a smooth and responsive interactive experience, and thus improving user satisfaction.

[0028] In one embodiment, an array of multiple microphones is configured inside the smart refrigerator. These microphones are distributed in different positions of the refrigerator to cover the largest possible pickup range and ensure that environmental noise can be captured from all angles. Such a design not only helps to obtain comprehensive noise information, but also enhances the sound collection capability in a specific direction through beamforming technology, further improving the signal-to-noise ratio. When the voice interaction action is initiated (for example, the user presses the voice button or says the wake-up word), the system will activate all microphones at the same time to start collecting sound data. At this time, in addition to paying attention to the user's voice commands, the system will also pay special attention to the noise conditions in the surrounding environment. The collected audio stream will be processed in two parts: one part is used for immediate voice command parsing; the other part is specifically used to analyze the characteristics of environmental noise.

[0029] Exemplarily, in the command voice signal adaptive noise reduction module 40, based on the environmental noise signal, the user interaction command voice signal after the primary noise reduction is adaptively noise-reduced to obtain the user interaction command voice signal after the secondary noise reduction. It should be understood that in a home environment, the type and intensity of background noise will change with time and place, including various types of noise such as range hoods, TV sounds, and conversation sounds. Traditional fixed parameter noise reduction methods are difficult to cope with such diverse and dynamically changing noise environments, which may lead to over-reduction (voice distortion) or under-reduction (noise residue), affecting the effectiveness of human-computer interaction in smart refrigerators and reducing the user's experience. Therefore, it is crucial to introduce adaptive noise reduction technology. It can dynamically adjust the noise reduction strategy according to the environmental noise information collected in real time to better adapt to the noise characteristics in different scenarios.

[0030] In particular, in the human-computer interaction system of the above-mentioned smart refrigerator, the user interaction command voice signal is subjected to noise reduction processing by applying a two-stage noise reduction strategy combining a frequency domain filter and an adaptive noise reduction technology, so as to achieve better noise reduction and enhancement effects of the user interaction command voice signal, which is conducive to extracting useful interaction commands and content semantics from the original audio data to perform corresponding control of the refrigerator. In other words, the above-mentioned human-computer interaction system of the smart refrigerator aims to solve the problem of voice recognition accuracy in complex environments by combining advanced signal processing technology and artificial intelligence algorithms. Specifically, the system first performs preliminary noise reduction processing using the data collected by the microphone array, and then further purifies the user interaction command voice signal adaptively based on the environmental noise information collected at the same time. This two-stage noise reduction process not only takes into account the basic physical level denoising, but also adds the understanding of contextual information to achieve better voice clarity restoration effect. Finally, the operation instructions are output by using voice recognition technology and command parsing to control the refrigerator. In this way, the noise reduction strategy can be dynamically adjusted through real-time analysis of the environmental noise characteristics, so as to more effectively perform noise reduction processing on the user interaction command voice signal, making it more efficient and accurate to extract useful information from the original audio data, thereby improving the accuracy of human-computer interaction and user satisfaction of the smart refrigerator in a more intelligent way.

[0031] In particular, the process of adaptively reducing the noise of the user interaction command voice signal after the primary noise reduction based on the environmental noise signal is crucial. Through the adaptive noise reduction technology, the system can more accurately estimate and remove the noise components in the background, thereby retaining a clearer user command voice signal. This directly improves the accuracy of the subsequent voice recognition stage and reduces the misrecognition or unrecognition caused by noise interference. Not only that, the sound environment in home spaces such as kitchens is very complex and constantly changing, including various types of sounds such as the working sound of household appliances and conversations. Traditional fixed parameter noise reduction methods are difficult to cope with this diversity. The adaptive noise reduction method can dynamically adjust the noise reduction strategy according to the environmental noise information collected in real time, and better adapt to the noise characteristics in different scenarios. Through this two-stage noise reduction processing, it can ensure that the user's commands can be effectively captured even in a noisy environment, providing a smooth and responsive interactive experience, thereby improving user satisfaction.

[0032] Specifically, in the process of adaptively denoising the user interaction instruction voice signal after primary noise reduction based on the environmental noise signal, the technical concept of the present application is to convert both the environmental noise signal and the user interaction instruction voice signal after primary noise reduction into a two-dimensional time-frequency graph, and then introduce an image processing and analysis algorithm based on artificial intelligence and deep learning at the back end to analyze the two-dimensional time-frequency graph of environmental noise and the two-dimensional time-frequency graph of voice signal, so as to capture the time-frequency features of environmental noise and the time-frequency features of voice signal, and dynamically memorize the time-frequency semantics of interactive instruction voice signal and optimize the expression based on the time-frequency semantics of environmental noise, so as to use the time-frequency interaction semantics of the two to characterize the time-frequency features of voice signal after adaptive noise reduction, and then obtain the user interaction instruction voice signal after secondary noise reduction by signal restoration. In this way, the environmental noise signal can be used to adaptively perform dynamic noise reduction processing on the user interaction instruction voice signal, so as to better adapt to the noise reduction requirements in different scenarios. In this way, a more intelligent refrigerator human-computer interaction processing can be achieved to ensure that the user's commands can be effectively captured even in a noisy environment, thereby helping to improve the responsive interactive experience and improve user satisfaction.

[0033] In one embodiment, Figure 3As shown, the command voice signal adaptive noise reduction module 40 includes: an environmental noise signal conversion unit 41, which is used to convert the environmental noise signal into a two-dimensional time-frequency diagram of environmental noise; an instruction voice signal conversion unit 42, which is used to convert the user interaction instruction voice signal after primary noise reduction into a two-dimensional time-frequency diagram of voice signal; a sound signal time-frequency feature extraction unit 43, which is used to extract sound signal time-frequency features from the two-dimensional time-frequency diagram of environmental noise and the two-dimensional time-frequency diagram of voice signal respectively to obtain an environmental noise time-frequency feature vector and a voice signal time-frequency feature vector; an environmental noise-voice signal feature interaction processing unit 44, which is used to perform feature interaction processing based on dynamic memory on the environmental noise time-frequency feature vector and the voice signal time-frequency feature vector to obtain the time-frequency features of the voice signal after adaptive noise reduction; a user interaction instruction generation unit 45 after noise reduction, which is used to generate the user interaction instruction voice signal after secondary noise reduction based on the time-frequency features of the voice signal after adaptive noise reduction.

[0034] Exemplarily, in the environmental noise signal conversion unit 41, the environmental noise signal is converted into a two-dimensional time-frequency diagram of environmental noise. It should be understood that environmental noise usually contains multiple frequency components, and these components will change over time. By converting the environmental noise signal into a two-dimensional time-frequency diagram, the time change (i.e., the development over time) and frequency distribution of the noise can be captured at the same time, thereby fully describing the dynamic characteristics of the noise. This is very important for distinguishing transient noise (such as sudden noises) from continuous noise (such as the sound of a fan). Two-dimensional time-frequency diagrams provide an intuitive way to observe and analyze patterns in noise signals. By processing these graphically represented data using a deep learning model, meaningful feature vectors can be extracted from complex background noise, thereby helping to design more effective noise reduction strategies.

[0035] In one embodiment, the environmental noise signal is converted into a two-dimensional time-frequency diagram of the environmental noise, including: first, the original audio data collected by the microphone array needs to be processed preliminarily, such as removing DC bias, normalization and other operations to ensure the consistency and stability of the input data. Then, the short-time Fourier transform (STFT) or continuous wavelet transform (CWT) is applied. Both methods can decompose a long time series signal into a series of spectral representations in a shorter time period. STFT is suitable for stationary noise sources, while CWT is more suitable for non-stationary signals because it can analyze signals at different scales. Specifically, STFT: select an appropriate window length and overlap rate to divide the signal into multiple short segments, and then calculate the Fourier transform for each segment to obtain its corresponding spectrum. Finally, the spectra of all segments are combined to form a two-dimensional matrix, in which the rows represent the position on the time axis and the columns correspond to the various frequency components. CWT: For each point, wavelet basis functions of different scales are used to convolve with the signal to generate a series of coefficient maps at different scales. These coefficient maps also constitute a two-dimensional structure that reflects the performance of the signal at different times and frequencies. Finally, the result of the above transformation can be visualized as a color image, where the color depth represents the amplitude. This forms the so-called "heat map" or "pseudo-color map", that is, a two-dimensional time-frequency map of environmental noise.

[0036] Exemplarily, in the command voice signal conversion unit 42, the user interaction command voice signal after primary noise reduction is converted into a two-dimensional time-frequency diagram of the voice signal. It should be understood that the two-dimensional time-frequency diagram can highlight the key features in the voice signal, such as the transition between phonemes, short sound pulses, etc. These features are crucial for understanding the voice content. By observing the frequency distribution in different time periods in the time-frequency diagram, it is easier to distinguish which parts are voice and which are unnecessary noise, thereby providing a basis for adaptive noise reduction. Modern deep learning technology is good at extracting complex patterns from images, so converting voice signals into a data format similar to an image (i.e., a two-dimensional time-frequency diagram) can make full use of existing image processing technology and network architecture to achieve efficient voice processing tasks.

[0037] Exemplarily, in the sound signal time-frequency feature extraction unit 43, the sound signal time-frequency feature extraction is performed on the two-dimensional time-frequency diagram of the environmental noise and the two-dimensional time-frequency diagram of the speech signal to obtain the time-frequency feature vector of the environmental noise and the time-frequency feature vector of the speech signal. It should be understood that, considering that for the sound signal, the time-frequency diagram provides information about the change of the frequency of the sound signal over time, and at the same time, this signal time-frequency feature can provide information about the difference between noise and interactive speech. Based on this, in order to be able to learn and extract key features that help distinguish noise from useful speech signals from the two-dimensional time-frequency diagram of the environmental noise and the two-dimensional time-frequency diagram of the speech signal, it is necessary to extract and capture the two-dimensional time-frequency diagram features of the two. Based on this, in the technical solution of the present application, the two-dimensional time-frequency diagram of the environmental noise and the two-dimensional time-frequency diagram of the speech signal are respectively input into the sound signal time-frequency feature extractor based on the GoogLeNet network to obtain the time-frequency feature vector of the environmental noise and the time-frequency feature vector of the speech signal. The time-frequency feature extractor for sound signals based on the GoogLeNet network can capture the semantics of the time-frequency features and the complex signal patterns in the two-dimensional time-frequency diagram of the environmental noise and the two-dimensional time-frequency diagram of the speech signal. In particular, it is worth mentioning that the design of the Inception module in the GoogLeNet architecture allows the simultaneous processing of receptive fields of different scales in the same layer, which means that it can capture local details and global structural information in the two-dimensional time-frequency diagram of the environmental noise and the two-dimensional time-frequency diagram of the speech signal. This is particularly useful for understanding subtle changes in sound signals, such as short noise pulses or transitions between speech phonemes.

[0038] In one embodiment, the sound signal time-frequency feature extraction unit is used to: input the two-dimensional time-frequency graph of the ambient noise and the two-dimensional time-frequency graph of the speech signal into the sound signal time-frequency feature extractor based on the GoogLeNet network to obtain the time-frequency feature vector of the ambient noise and the time-frequency feature vector of the speech signal. Here, one of the most notable features of GoogLeNet is the introduction of the Inception module. This module processes receptive fields of different scales in one layer at the same time, by using convolution kernels of 1x1, 3x3, and 5x5 sizes and 3x3 maximum pooling operations in parallel, and then splicing their results together as the output of the layer. The advantage of doing this is that the network can automatically learn the best spatial aggregation method, and increase the network width without significantly increasing the computational cost. In addition, in order to alleviate the problem of gradient disappearance that may occur during deep network training, GoogLeNet adds two auxiliary classifier branches in the middle of the network. These branches are connected to the middle layer and participate in the loss function calculation together with the main classifier, providing additional supervision signals, which helps to accelerate the convergence process. Only the results of the main classifier are considered in the final prediction.

[0039] Unlike traditional CNNs that use a large number of fully connected layers to extract features, GoogLeNet uses global average pooling layers to replace the functions of some fully connected layers. This not only reduces the number of parameters, but also reduces the risk of overfitting, while also improving the generalization ability of the model. GoogLeNet contains a total of 22 layers (more if the layers inside the Inception module are counted), but its number of parameters is far less than other top models at the time. This is achieved through a carefully designed network topology, including the above-mentioned Inception module and global average pooling and other technical means.

[0040] Due to its high efficiency and accuracy, GoogLeNet is widely used in various computer vision tasks, especially in image classification, object detection and other fields. For example, in the human-computer interaction system of smart refrigerators, GoogLeNet can be used to extract features from the two-dimensional time-frequency graph of environmental noise and the two-dimensional time-frequency graph of speech signals. Specifically, when useful features need to be extracted from audio data, the two-dimensional time-frequency graph of environmental noise and the two-dimensional time-frequency graph of speech signals can be input into the sound signal time-frequency feature extractor based on the GoogLeNet network respectively. GoogLeNet can capture the local details and global structural information in these images, thereby helping to distinguish the key features of noise and useful speech signals.

[0041] Exemplarily, in the environmental noise-speech signal feature interaction processing unit 44, the environmental noise time-frequency feature vector and the speech signal time-frequency feature vector are subjected to feature interaction processing based on dynamic memory to obtain the time-frequency feature of the speech signal after adaptive noise reduction. It should be understood that the environmental noise time-frequency feature vector and the speech signal time-frequency feature vector respectively contain two-dimensional time-frequency feature information about the environmental noise and the user interaction instruction speech signal. However, in the actual interaction scenario, the environmental noise is complex and changeable, and has different categories. In such a complex and changeable environment, the relationship between noise and speech is nonlinear and changes with time and frequency. In order to effectively interact and fuse the time-frequency semantics of the environmental noise and the time-frequency semantics of the interactive instruction speech signal, so as to extract a clear speech signal from the noise, in the technical solution of the present application, the environmental noise time-frequency feature vector and the speech signal time-frequency feature vector are further subjected to feature interaction processing based on dynamic memory to obtain the time-frequency feature of the speech signal after adaptive noise reduction.

[0042] In one embodiment, Figure 4As shown, the environmental noise-speech signal feature interaction processing unit 44 includes: a collaborative feature extraction subunit 441, which is used to perform collaborative feature extraction on the environmental noise time-frequency feature vector and the speech signal time-frequency feature vector to obtain an implicit collaborative coding vector between the speech signal-environmental noise time-frequency features; a feature attention modulation optimization subunit 442, which is used to perform feature attention modulation optimization on the speech signal time-frequency feature vector and the environmental noise time-frequency feature vector based on the implicit collaborative coding vector between the speech signal-environmental noise time-frequency features to obtain an optimized speech signal time-frequency feature vector and an optimized environmental noise time-frequency feature vector; a feature linear interaction processing subunit 443, which is used to input the optimized speech signal time-frequency feature vector and the optimized environmental noise time-frequency feature vector into a feature linear interaction network to obtain an adaptive denoised speech signal time-frequency feature vector as the adaptive denoised speech signal time-frequency feature.

[0043] Exemplarily, in the collaborative feature extraction subunit 441, collaborative feature extraction is performed on the ambient noise time-frequency feature vector and the speech signal time-frequency feature vector to obtain an implicit collaborative coding vector between the speech signal and ambient noise time-frequency features. Specifically, the process can be expressed as follows: ;in, and are respectively the time-frequency feature vector of the speech signal and the time-frequency feature vector of the ambient noise, and They are positional addition, positional subtraction, and positional dot multiplication. Indicates cascade processing, represents the point convolution layer, express Activation function, It is the implicit collaborative coding vector between the time-frequency features of speech signal and environmental noise.

[0044] That is, by extracting the time-frequency feature vectors of environmental noise and speech signals in a collaborative way, we can capture more complex interaction patterns between the two. This interaction pattern contains key clues about how to distinguish useful speech information from interfering noise, allowing subsequent noise reduction algorithms to more accurately remove background noise while retaining and enhancing the user's voice commands.

[0045] Exemplarily, in the feature attention modulation optimization subunit 442, based on the implicit collaborative coding vector between the speech signal and the environmental noise time-frequency features, the speech signal time-frequency feature vector and the environmental noise time-frequency feature vector are respectively subjected to feature attention modulation optimization to obtain an optimized speech signal time-frequency feature vector and an optimized environmental noise time-frequency feature vector. Specifically, the process can be expressed as follows: ; in, is the implicit collaborative coding vector between the time-frequency features of speech signal and environmental noise, represents a 1×1 convolution operation, is the dynamic key vector of the time-frequency characteristics of speech signal-environmental noise, and They represent the speech signal time-frequency query weight matrix and the speech signal time-frequency query bias vector respectively, and Respectively represent the speech signal time-frequency value weight matrix and the speech signal time-frequency value bias vector, is matrix multiplication, and are the speech signal time-frequency query feature vector and the speech signal time-frequency value feature vector respectively. is the vector transpose, for Length, represents the normalized exponential function, is the optimized time-frequency feature vector of speech signal, and They represent the ambient noise time-frequency query weight matrix and the ambient noise time-frequency query bias vector respectively, and They represent the ambient noise time-frequency value weight matrix and the ambient noise time-frequency value bias vector respectively, and are the environmental noise time-frequency query feature vector and the environmental noise time-frequency value feature vector respectively. for Length, is the time-frequency feature vector of environmental noise after optimization.

[0046] In one embodiment, the feature attention modulation optimization subunit is used to: write the implicit collaborative coding vector between the speech signal and environmental noise time-frequency features into a dynamic memory unit to obtain a speech signal-environmental noise time-frequency feature dynamic key vector; extract the speech signal-environmental noise time-frequency feature dynamic key vector from the dynamic memory unit, and input the speech signal time-frequency feature vector and the speech signal-environmental noise time-frequency feature dynamic key vector into a feature attention modulation module based on a first converter structure to obtain the optimized speech signal time-frequency feature vector; extract the speech signal-environmental noise time-frequency feature dynamic key vector from the dynamic memory unit, and input the environmental noise time-frequency feature vector and the speech signal-environmental noise time-frequency feature dynamic key vector into a feature attention modulation module based on a second converter structure to obtain the optimized environmental noise time-frequency feature vector.

[0047] Specifically, the process of feature interaction processing based on dynamic memory can effectively capture the implicit synergistic relationship between the time-frequency feature vector of the ambient noise and the time-frequency feature vector of the speech signal, and use the implicit synergistic relationship between the time-frequency semantics of the ambient noise and the time-frequency semantics of the speech signal as a dynamic key vector to construct the same attention space to perform attention modulation optimization on the time-frequency feature vector of the ambient noise and the time-frequency feature vector of the speech signal in real time, thereby emphasizing the important information therein and suppressing noise and redundant information. The feature attention modulation module based on the converter structure adopts a self-attention mechanism, allowing the model to consider the influence of all other elements when processing each input element. This global perspective is particularly important for understanding long-distance dependencies. Through feature modulation based on query attention, the expression of key information in the feature vector can be enhanced while reducing the influence of noise and irrelevant information. For example, at a specific time point, some frequency components may be noise, while at another time point, they may be useful speech signals. The self-attention mechanism can help the model correctly allocate attention at different time points, thereby better separating noise and speech. This helps to highlight the useful parts of the speech signal and improve speech clarity.

[0048] Exemplarily, in the feature linear interaction processing subunit 443, the optimized speech signal time-frequency feature vector and the optimized ambient noise time-frequency feature vector are input into the feature linear interaction network to obtain the adaptive noise reduction speech signal time-frequency feature vector as the adaptive noise reduction speech signal time-frequency feature. Specifically, the process can be expressed as follows: ;in, is the optimized time-frequency feature vector of speech signal, is the time-frequency feature vector of the optimized ambient noise, is the weight hyperparameter, is the time-frequency feature vector of the speech signal after adaptive noise reduction.

[0049] That is, the optimized time-frequency feature vector of environmental noise and the optimized time-frequency feature vector of speech signal are feature aligned and linearly fused. During the fusion process, the weights are dynamically optimized by introducing trainable weight hyperparameters, so that the time-frequency features of environmental noise can be used to optimize the expression of time-frequency features of speech signals, thereby generating an adaptive denoised speech signal time-frequency feature vector optimized by adaptive denoising modulation, providing high-quality input for subsequent speech recognition.

[0050] Exemplarily, in the noise reduction user interaction instruction generating unit 45, the secondary noise reduction user interaction instruction voice signal is generated based on the time-frequency characteristics of the adaptive noise reduction voice signal. It should be understood that the data processed by the feature extractor is usually in the form of feature vectors. Although these feature vectors contain important acoustic information, they do not directly correspond to audible sound waveforms. In order to be understood and processed by the subsequent speech recognition model, these feature vectors must be converted back to audio signals in the time domain. Therefore, in one embodiment, the noise reduction user interaction instruction generating unit is used to: perform signal restoration on the adaptive noise reduction voice signal time-frequency feature vector to obtain the secondary noise reduction user interaction instruction voice signal. By performing signal restoration on the adaptive noise reduction voice signal time-frequency feature vector, a clearer and lower noise noise reduction user interaction instruction voice signal can be generated. This process helps to remove or reduce the impact of background noise while retaining and enhancing the user's voice content components, thereby improving the quality of the final voice signal. In this way, the environmental noise signal can be used to adaptively perform dynamic noise reduction processing on the user interaction command voice signal, so as to better adapt to the noise reduction needs in different scenarios. In this way, more intelligent refrigerator human-computer interaction processing can be achieved to ensure that the user's commands can be effectively captured even in noisy environments, which in turn helps to improve the responsive interaction experience and enhance user satisfaction.

[0051] In a specific embodiment, a decoder is used to restore the time-frequency feature vector of the adaptive denoised speech signal to obtain the user interaction instruction speech signal after the secondary denoising. Specifically, first, an input layer is used to receive the time-frequency feature vector of the adaptive denoised speech signal as input. Then, a deconvolution layer is used to gradually restore the spatial resolution through a transposed convolution operation and introduce new detail information. These layers can help expand the spatial dimension of the input feature map while retaining important frequency information. Next, batch normalization and activation functions are applied after each deconvolution layer to accelerate the training process and stabilize the output distribution, and introduce nonlinearity so that the model can learn more complex mapping relationships. Then, in order to better capture long-term dependencies in time series, several GRU layers are added on top of the deconvolution layer. GRU is a simplified version of LSTM (Long Short-Term Memory Network), which can effectively process sequence data and has low computational cost. Finally, a fully connected layer is used to map the high-dimensional features back to the dimensions of the original audio waveform to obtain the user interaction instruction speech signal after the secondary denoising.

[0052] In the technical solution of the present application, the time-frequency feature vector of the ambient noise and the time-frequency feature vector of the speech signal respectively represent the time-frequency features of the ambient noise signal and the time-frequency features of the user interaction command speech signal after primary noise reduction. In the process of performing attention interaction between features based on the dynamic memory mechanism, the randomness of the ambient noise signal will cause the fine-grained interaction structure in the dynamic interaction process to be unbalanced, which will affect the signal quality restored by the decoded signal of the speech signal time-frequency feature vector after adaptive noise reduction.

[0053] Based on this, in a preferred embodiment of the present application, the time-frequency feature vector of the adaptive denoising speech signal is restored to obtain the secondary denoising user interaction command speech signal, including: performing feature manifold optimization on the time-frequency feature vector of the adaptive denoising speech signal to obtain an optimized time-frequency feature vector of the adaptive denoising speech signal; performing signal restoration on the optimized time-frequency feature vector of the adaptive denoising speech signal to obtain the secondary denoising user interaction command speech signal.

[0054] Specifically, the process of feature manifold optimization includes: calculating the L1 norm of the time-frequency feature vector of the adaptive denoised speech signal to obtain the time-frequency feature topological parameter of the first speech signal, and calculating its L2 norm to obtain the time-frequency feature topological parameter of the second speech signal; ; in, represents each eigenvalue of the time-frequency eigenvector of the speech signal after adaptive noise reduction, represents the modulus length of the time-frequency feature vector of the speech signal after the adaptive noise reduction, represents the time-frequency characteristic topological parameter of the first speech signal, Represents the time-frequency feature topological parameter of the second speech signal.

[0055] The eigenvalues ​​of each position of the adaptive denoised speech signal time-frequency feature vector are multiplied by the first speech signal time-frequency feature topological parameter and the second speech signal time-frequency feature topological parameter to generate the corresponding speech signal time-frequency feature first-order topological reference and speech signal time-frequency feature second-order topological reference, which are expressed as: ; in, Represents eigenvalue The corresponding first-order topological reference of the time-frequency characteristics of the speech signal, Represents eigenvalue The corresponding second-order topological reference of the time-frequency characteristics of the speech signal.

[0056] The eigenvalues ​​at each position of the adaptive denoised speech signal time-frequency feature vector are respectively multiplied by the modulus length and the square root of the adaptive denoised speech signal time-frequency feature vector to generate a first-order domain transformation parameter of the speech signal time-frequency feature and a second-order domain transformation parameter of the speech signal time-frequency feature, which are expressed as: ; in, Represents eigenvalue The corresponding first-order domain transformation parameters of the speech signal time-frequency characteristics, Represents eigenvalue The corresponding second-order domain transformation parameters of the speech signal time-frequency characteristics.

[0057] The first-order topological reference of the speech signal time-frequency feature is divided by the offset between the first speech signal time-frequency feature topological parameter and the first-order domain transformation parameter of the speech signal time-frequency feature to obtain the first-order modulation coefficient of the speech signal time-frequency feature, which is expressed as: ;in, Represents eigenvalue The corresponding first-order modulation coefficient of the speech signal time-frequency characteristics.

[0058] The second-order topological reference of the speech signal time-frequency feature is divided by the offset between the second speech signal time-frequency feature topological parameter and the second-order domain transformation parameter of the speech signal time-frequency feature to obtain the second-order modulation coefficient of the speech signal time-frequency feature, which is expressed as: ;in, Represents eigenvalue The corresponding second-order modulation coefficient of the speech signal time-frequency characteristics.

[0059] The weighted aggregation between the first-order modulation coefficient of the speech signal time-frequency feature and the second-order modulation coefficient of the speech signal time-frequency feature is calculated to obtain each eigenvalue of the optimized adaptive noise-reduced speech signal time-frequency feature vector, which is expressed as: ;in, represents a vector composed of the first-order modulation coefficients of the time-frequency characteristics of the speech signal, represents a vector composed of the second-order modulation coefficients of the time-frequency characteristics of the speech signal, represents the weighted hyperparameter, It means adding by position. It means point multiplication by position. Represents the optimized time-frequency feature vector of the speech signal after adaptive denoising.

[0060] That is, for the geometric topological features of the attribute clusters of the time-frequency feature vectors of the adaptive denoised speech signal in the multidimensional embedding domain, the normalized spatial topological representation of the time-frequency feature vectors of the adaptive denoised speech signal is used as the reference field of view to implement the scale-driven domain transformation of each attribute value, and complete the construction of the regional attention modulation mechanism. In this way, it is ensured that the time-frequency feature vectors of the adaptive denoised speech signal maintain spatial mapping stability during the attribute field interaction process, thereby enhancing the stable convergence and extrapolation performance of the generative morphological fitting of its attribute clusters in a heterogeneous semantic topological environment, and ultimately improving the signal restoration quality of the user interaction command speech signal after the secondary denoising.

[0061] Exemplarily, in the interactive instruction voice recognition module 50, the user interactive instruction voice signal after the secondary noise reduction is sent to the voice recognition model to obtain the user interactive instruction text description. It should be understood that the voice signal after the primary noise reduction and adaptive noise reduction processing has significantly reduced the interference of background noise, which makes the voice signal clearer, thereby improving the input quality of the voice recognition model. High-quality input helps to reduce the possibility of misrecognition and ensure that the system can accurately understand the user's intention. Recognition through optimized voice signals can provide smoother and faster response time, reduce delays or false feedback caused by noise, and thus improve user satisfaction. Even in a noisy environment, users can obtain a stable and reliable interactive experience. Modern voice recognition technology is not limited to simple keyword matching, but can also parse complex natural language expressions. By providing clean audio input to the voice recognition model, the system can better capture the nuances in user commands, such as multi-step instructions or multi-condition queries, and provide users with more intelligent services.

[0062] In one embodiment, the speech recognition model is DeepSpeech. It should be understood that DeepSpeech is an end-to-end speech recognition system based on deep learning, which uses recurrent neural networks (RNN) and convolutional neural networks (CNN) to directly map audio waveforms to text. In a specific embodiment, a suitable installation method is selected according to the target platform (such as installing a Python package through pip). Download and load the pre-trained DeepSpeech model file (.pbmm format). Using the provided Python API or other programming interface, the user interaction instruction voice signal after the secondary noise reduction is passed to the DeepSpeech model for recognition to obtain a text description of the user interaction instruction. The returned text description is obtained as the basis for command parsing.

[0063] Exemplarily, in the command parsing module 60, the text description of the user interaction instruction is parsed to obtain the command content parsing result. It should be understood that the text description generated by the speech recognition model only converts the user's speech into text form, but it does not directly tell the system which specific operations should be performed. Through command parsing, the real needs of users can be deeply understood, including the information they want to query, the functions they want to set, or the actions they want to trigger. Modern smart home devices can not only respond to simple keywords or phrases, but also understand and execute complex multi-step instructions or multi-conditional queries. For example, "Please adjust the temperature to 4 degrees and tell me what food has the shortest remaining shelf life." This type of complex instruction requires the use of advanced natural language processing technology to accurately parse and decompose into a series of executable operations. Through a strict command parsing process, the validity of user input can be verified to prevent misoperation or illegal commands from being executed. In addition, it can help filter out ambiguous or incomplete requests to ensure that only clear and reasonable commands are passed to the control module. Based on the user's historical behavior and personal preferences, command parsing can also provide customized suggestions and services. For example, recommend appropriate food storage methods based on the user's eating habits, or automatically adjust the working mode of the refrigerator to save energy.

[0064] In one embodiment, the user interaction instruction text description is subjected to command parsing to obtain a command content parsing result. The user interaction instruction text description is subjected to command parsing to obtain a command content parsing result, including: first, using a word segmentation tool to segment the continuous text into words or phrase units, which helps to understand the sentence structure in a more fine-grained manner later. Dependency syntactic analysis, named entity recognition (NER), sentiment analysis and other technologies are applied to capture key information in the text, such as subject-verb-object relationship, specific nouns (such as food names), verb intentions, etc. Then, by training a machine learning model or a rule engine, the user's main intention is identified, such as query, setting, starting a certain function, etc. These models can be trained based on a large amount of annotated data to better adapt to different types of user input. A set of possible command modes and their corresponding execution logic are predefined to form a command template library. Each template contains a specific slot for filling in specific parameter values ​​extracted from user input. The text fragment processed by NLP is matched with the command template to find the template that best matches the current input, and the relevant parameters are filled in the corresponding position. For example, for the command "set the temperature to X degrees", X is the parameter to be extracted.

[0065] In other embodiments, considering that the user may issue multiple related commands in succession, the system needs to maintain a certain context memory in order to understand the connection between the previous and subsequent commands. This can be achieved by introducing a dialogue management mechanism to track the dialogue history and dynamically update the internal state. When encountering ambiguous or multiple interpretation possibilities of input, the system should have the ability to ask for clarification, provide the user with a list of options or ask for further explanation, so as to ensure that the final parsing result is correct. Combined with refrigerator-related professional knowledge (such as food types, shelf life, etc.), the ability to understand specific terms is enhanced. This not only improves the accuracy of parsing, but also provides users with more professional and considerate services.

[0066] Exemplarily, in the refrigerator operation control feedback module 70, based on the command content parsing result, the operation instruction is called to control the refrigerator and the operation result is fed back. It should be understood that the clear instructions obtained through command parsing can guide the system to accurately call the corresponding operation instructions to control the functions of the refrigerator. This not only improves the response accuracy of the system, but also avoids operational errors caused by misunderstandings or incorrect parsing. When the user issues a voice command, he can immediately see or hear the system's specific response to his request, which enhances the immediacy and intuitiveness of the interaction. This timely feedback makes users feel that their needs are valued and that the system is working properly.

[0067] In one embodiment, the refrigerator operation control feedback module is used to: based on the command content parsing result, call the operation instruction from the operation instruction library to control the refrigerator; and feedback the operation result through a voice feedback signal. Specifically, when constructing the operation instruction library, a set of standardized command formats are created for various functions of the refrigerator, including but not limited to power on and off, temperature setting, lighting control, food classification and shelf life management. Each standard command corresponds to one or more API interfaces, which are responsible for actually performing specific tasks to ensure that all possible operations have a clear mapping relationship for subsequent calls. According to the parsing result, the operation instruction that best matches the current command description is searched in the operation instruction library. If there are multiple possible choices, the best match is selected; if there is ambiguity, the user is asked for further clarification. Once the specific operation instruction is determined, the corresponding API interface or function can be called to execute the command, such as adjusting the temperature setting, querying the inventory status, etc. During the execution of the operation, the system will continuously monitor the state changes of the refrigerator to ensure that the operation is completed smoothly. If a problem is encountered (such as hardware failure, insufficient authority, etc.), the abnormal situation will be captured in time and appropriate measures will be taken. Whether it is successful or failed, concise and clear feedback information should be generated to inform the user. For successful operations, a simple confirmation can be given; for failures, the reasons need to be explained and solutions provided. Feedback can be provided in a variety of ways, including voice feedback: playing voice messages directly through the built-in speaker, such as "The temperature has been adjusted to 4 degrees", "Cannot find the food you mentioned, please check whether the name is correct"; or graphical interface feedback: updating the display content on the refrigerator display screen to present the operation results in a visual form, such as icon changes, text prompts, etc.; in addition, if the user is not near the refrigerator, notifications can be sent through mobile applications or other remote terminals to keep the information synchronized. In this way, by calling the operation instructions based on the command content analysis results to control the refrigerator and feedback the operation results, not only the user experience is improved, but also the adaptability and reliability of the system are enhanced.

[0068] In summary, the human-computer interaction system of the smart refrigerator according to the embodiment of the present application is explained, which uses the data collected by the microphone array to perform preliminary noise reduction processing, and then adaptively further purifies the user interaction command voice signal according to the environmental noise information collected at the same time. This two-stage noise reduction process not only takes into account the basic physical level denoising, but also incorporates the understanding of contextual information to achieve better voice clarity restoration effect. Finally, the operation instructions are output by using voice recognition technology and command parsing to control the refrigerator. In this way, the noise reduction strategy can be dynamically adjusted by real-time analysis of the environmental noise characteristics, so as to more effectively perform noise reduction processing on the user interaction command voice signal, making it more efficient and accurate to extract useful information from the original audio data, and improving the accuracy and user satisfaction of the human-computer interaction of the smart refrigerator in a more intelligent way.

[0069] Figure 5 Schematic flow chart of the human-computer interaction method of the smart refrigerator of the embodiment of the present application. Figure 5 As shown, the human-computer interaction method of the smart refrigerator includes: S1, after receiving a voice interaction action to start, using the built-in microphone array of the refrigerator to collect a user interaction command voice signal; S2, performing primary noise reduction on the user interaction command voice signal to obtain a user interaction command voice signal after primary noise reduction; S3, using the microphone array to collect an environmental noise signal; S4, based on the environmental noise signal, performing adaptive noise reduction on the user interaction command voice signal after primary noise reduction to obtain a user interaction command voice signal after secondary noise reduction; S5, sending the user interaction command voice signal after secondary noise reduction to a voice recognition model to obtain a user interaction command text description; S6, performing command parsing on the user interaction command text description to obtain a command content parsing result; S7, based on the command content parsing result, calling an operation command to control the refrigerator and feeding back the operation result.

[0070] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned human-computer interaction method of the smart refrigerator have been referred to above. Figures 1 to 4 The human-computer interaction system of the smart refrigerator is introduced in detail, and therefore, its repeated description will be omitted.

[0071] An embodiment of the present application further provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the methods in the above embodiments of the present application.

[0072] An embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer implements the methods in the above embodiments of the present application.

[0073] An embodiment of the present application also provides a chip, including a circuit, for executing the methods in the above embodiments of the present application.

[0074] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0075] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is a kind of association relationship that describes associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In this application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0076] The prefixes such as "first" and "second" used in the embodiments of the present application are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers used to distinguish description objects in the embodiments of the present application does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0077] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0078] In the various embodiments of the present application, unless otherwise specified or logically conflicting, the terms and / or descriptions between the various embodiments are consistent and may be referenced to each other, and the technical features in different embodiments may be combined to form new embodiments according to their inherent logical relationships.

[0079] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0080] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0081] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A human-computer interaction system for a smart refrigerator, characterized in that: include: A user interaction command voice collection module is used to collect user interaction command voice signals using a built-in microphone array of the refrigerator after receiving a voice interaction start action; A command voice signal primary noise reduction module, used for performing primary noise reduction on the user interaction command voice signal to obtain a user interaction command voice signal after primary noise reduction; An environmental noise signal acquisition module, used to acquire environmental noise signals using the microphone array; A command voice signal adaptive noise reduction module, used for adaptively reducing the noise of the user interaction command voice signal after primary noise reduction based on the environmental noise signal to obtain a user interaction command voice signal after secondary noise reduction; An interactive command voice recognition module is used to send the user interactive command voice signal after secondary noise reduction to a voice recognition model to obtain a text description of the user interactive command; a command parsing module is used to perform command parsing on the user interactive command text description to obtain a command content parsing result; and a refrigerator operation control feedback module is used to call an operation command to control the refrigerator and feedback the operation result based on the command content parsing result.

2. The human-computer interaction system of the smart refrigerator according to claim 1, characterized in that: The action of starting voice interaction is that the user presses a voice button or speaks a wake-up word.

3. The human-computer interaction system of the smart refrigerator according to claim 2, characterized in that: The command voice signal primary noise reduction module is used to: use a frequency domain filter to perform primary noise reduction on the user interaction command voice signal to remove high-frequency noise of the user interaction command voice signal to obtain the user interaction command voice signal after primary noise reduction.

4. The human-computer interaction system of the smart refrigerator according to claim 3, characterized in that: The command voice signal adaptive noise reduction module includes: an environmental noise signal conversion unit, which is used to convert the environmental noise signal into a two-dimensional time-frequency diagram of environmental noise; a command voice signal conversion unit, which is used to convert the user interaction command voice signal after primary noise reduction into a two-dimensional time-frequency diagram of voice signal; a sound signal time-frequency feature extraction unit, which is used to extract sound signal time-frequency features from the two-dimensional time-frequency diagram of environmental noise and the two-dimensional time-frequency diagram of voice signal respectively to obtain an environmental noise time-frequency feature vector and a voice signal time-frequency feature vector; an environmental noise-voice signal feature interaction processing unit, which is used to perform feature interaction processing based on dynamic memory on the environmental noise time-frequency feature vector and the voice signal time-frequency feature vector to obtain the time-frequency features of the voice signal after adaptive noise reduction; and a user interaction command generation unit after noise reduction, which is used to generate the user interaction command voice signal after secondary noise reduction based on the time-frequency features of the voice signal after adaptive noise reduction.

5. The human-computer interaction system of the smart refrigerator according to claim 4, characterized in that: The sound signal time-frequency feature extraction unit is used to: input the two-dimensional time-frequency diagram of the environmental noise and the two-dimensional time-frequency diagram of the speech signal into a sound signal time-frequency feature extractor based on the GoogLeNet network to obtain the time-frequency feature vector of the environmental noise and the time-frequency feature vector of the speech signal.

6. The human-computer interaction system of the smart refrigerator according to claim 5, characterized in that: The environmental noise-speech signal feature interaction processing unit includes: a collaborative feature extraction subunit, which is used to perform collaborative feature extraction on the environmental noise time-frequency feature vector and the speech signal time-frequency feature vector to obtain an implicit collaborative coding vector between the speech signal-environmental noise time-frequency features; a feature attention modulation optimization subunit, which is used to perform feature attention modulation optimization on the speech signal time-frequency feature vector and the environmental noise time-frequency feature vector based on the implicit collaborative coding vector between the speech signal-environmental noise time-frequency features to obtain an optimized speech signal time-frequency feature vector and an optimized environmental noise time-frequency feature vector; a feature linear interaction processing subunit, which is used to input the optimized speech signal time-frequency feature vector and the optimized environmental noise time-frequency feature vector into a feature linear interaction network to obtain an adaptive denoised speech signal time-frequency feature vector as the adaptive denoised speech signal time-frequency feature.

7. The human-computer interaction system of the smart refrigerator according to claim 6, characterized in that: The feature attention modulation optimization subunit is used to: write the implicit collaborative coding vector between the speech signal and environmental noise time-frequency features into a dynamic memory unit to obtain a speech signal-environmental noise time-frequency feature dynamic key vector; extract the speech signal-environmental noise time-frequency feature dynamic key vector from the dynamic memory unit, and input the speech signal time-frequency feature vector and the speech signal-environmental noise time-frequency feature dynamic key vector into a feature attention modulation module based on a first converter structure to obtain the optimized speech signal time-frequency feature vector; extract the speech signal-environmental noise time-frequency feature dynamic key vector from the dynamic memory unit, and input the environmental noise time-frequency feature vector and the speech signal-environmental noise time-frequency feature dynamic key vector into a feature attention modulation module based on a second converter structure to obtain the optimized environmental noise time-frequency feature vector.

8. The human-computer interaction system of the smart refrigerator according to claim 7, characterized in that: The post-noise reduction user interaction instruction generating unit is used to perform signal restoration on the time-frequency feature vector of the adaptive post-noise reduction speech signal to obtain the secondary post-noise reduction user interaction instruction speech signal.

9. The human-computer interaction system of the smart refrigerator according to claim 8, characterized in that: The refrigerator operation control feedback module is used to: based on the command content analysis result, call the operation instruction from the operation instruction library to control the refrigerator; and feed back the operation result through a voice feedback signal.

10. A human-computer interaction method for a smart refrigerator, characterized in that: include: After receiving the voice interaction start action, the microphone array built into the refrigerator is used to collect the user interaction command voice signal; Performing primary noise reduction on the user interaction instruction voice signal to obtain a user interaction instruction voice signal after primary noise reduction; Collecting environmental noise signals using the microphone array; Based on the environmental noise signal, adaptively denoising the primary noise-reduced user interaction instruction voice signal to obtain a secondary noise-reduced user interaction instruction voice signal; Sending the secondary noise-reduced user interaction command voice signal to a voice recognition model to obtain a user interaction command text description; performing command parsing on the user interaction command text description to obtain a command content parsing result; Based on the analysis result of the command content, the operation instruction is called to control the refrigerator and the operation result is fed back.

Images