Speech interaction method based on multi-modal noise suppression
By combining a microphone and an image acquisition device, and using a large language model for multimodal noise suppression, the problem of traditional methods being unable to distinguish target speech from noise in complex environments is solved, and higher speech recognition accuracy is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANHE COLLEGE GUANGDONG POLYTECHNIC NORMAL UNIV
- Filing Date
- 2026-01-30
- Publication Date
- 2026-06-02
AI Technical Summary
Traditional noise reduction algorithms struggle to dynamically distinguish between target speech and noise in complex environments, leading to speech distortion or information loss.
By combining microphone-collected speech data with image acquisition device-captured lip movement video, feature extraction and matching are performed using a large language model to achieve multimodal noise suppression.
It improves the accuracy of speech recognition and avoids speech distortion and information loss, especially in complex and noisy environments.
Smart Images
Figure CN122135696A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of voice extraction technology, and in particular to a voice interaction method based on multimodal noise suppression. Background Technology
[0002] The problem of severe noise interference stems from the limitations of traditional noise reduction algorithms. In complex environments, such as noisy streets or multi-person conversations, background noise often exhibits non-stationary, high-frequency, and diverse characteristics. Currently, traditional methods mainly rely on spectral analysis or fixed threshold filtering, making it difficult to dynamically distinguish the complex superposition of target speech and noise. For example, when the user's speech overlaps with the background noise frequency band, the algorithm may over-suppress the speech signal, leading to speech distortion or information loss.
[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention
[0004] The main objective of this invention is to provide a voice interaction method based on multimodal noise suppression, aiming to improve the accuracy of speech recognition. To achieve the above objective, this invention provides a voice interaction method based on multimodal noise suppression, which includes the following steps: The microphone is controlled to collect voice data in the dialogue scene, and the image acquisition device is controlled to capture video of lip movements in the dialogue scene; The data matching results are determined based on the speech data, the lip movement video, and the preset large language model. The data matching results include: time matching results and sound source matching results. The denoising result is determined based on the data matching result and the speech data.
[0005] Optionally, the step of determining the data matching result based on the speech data, the lip movement video, and the preset large language model includes: The speech data and the lip movement video are input into the preset large language model for feature extraction to obtain speech feature vectors and visual feature vectors. The time matching result is determined based on the feature distribution patterns of the speech feature vector and the visual feature vector; The sound source matching result is determined based on the lip state information in the visual feature vector and the phoneme data in the speech feature vector.
[0006] Optionally, the step of determining the denoising result based on the data matching result and the speech data includes: Based on the time matching result, the speech data and the lip movement video are time-stamp aligned to obtain aligned multimodal data; Based on the sound source matching results, the target speech stream of the target speaker is located from the aligned multimodal data; The target speech stream is subjected to background noise suppression processing to obtain the denoising result.
[0007] Optionally, the step of performing background noise suppression processing on the target speech stream to obtain the denoising result includes: Lip shape feature data are determined based on the lip movement video; When the signal-to-noise ratio of the target speech stream is lower than a preset threshold, the missing phonemes in the target speech stream are inferred based on the lip shape feature data to obtain an enhanced speech stream. The enhanced speech stream is used as the denoising result.
[0008] Optionally, the step of performing auxiliary inference on missing phonemes in the target speech stream based on the lip shape feature data to obtain the enhanced speech stream includes: The visual phoneme sequence is determined based on the lip shape feature data; Calculate the probability distribution of the association between the visual phoneme sequence and the residual audio features in the target speech stream; The predictive compensation features for missing phonemes are determined based on the aforementioned correlation probability distribution; The target speech stream is reconstructed based on the predicted compensation features to obtain the enhanced speech stream.
[0009] Optionally, after the step of determining the denoising result based on the data matching result and the speech data, the method further includes: The text to be recognized is determined based on the denoising results; The target text result is determined based on the identified text and the preset dialect adaptive module; The step of determining the target text result based on the recognized text and the preset dialect adaptation module includes: Based on the identified text and a small number of dialect samples, the fine-tuned model parameters are determined using a Few-shot learning strategy, and the target text result is determined based on the fine-tuned model parameters.
[0010] Optionally, after the step of determining the target text result based on the recognized text and the preset dialect adaptation module, the method further includes: Contextual information is determined based on the target text result and the dialogue history; Based on the context information, the user's intent is determined through the semantic reasoning capabilities of the preset large language model.
[0011] Furthermore, to achieve the above objectives, the present invention also provides a voice interaction device based on multimodal noise suppression, the device comprising: The control module is used to control the microphone to collect voice data in the dialogue scenario and to control the image acquisition device to capture video of lip movements in the dialogue scenario. The alignment module is used to determine the data matching result based on the speech data, the lip movement video, and the preset large language model. The data matching result includes: time matching result and sound source matching result. A noise reduction module is used to determine the noise reduction result based on the data matching result and the speech data.
[0012] Furthermore, to achieve the above objectives, the present invention also provides a voice interaction device based on multimodal noise suppression. The voice interaction device based on multimodal noise suppression includes: a memory, a processor, and a voice interaction program based on multimodal noise suppression stored in the memory and executable on the processor. The voice interaction program based on multimodal noise suppression is configured to implement the steps of the voice interaction method based on multimodal noise suppression described above.
[0013] In addition, to achieve the above objectives, the present invention also provides a storage medium storing a voice interaction program based on multimodal noise suppression, wherein when the voice interaction program based on multimodal noise suppression is executed by a processor, it implements the steps of the voice interaction method based on multimodal noise suppression described above.
[0014] This invention proposes a voice interaction method based on multimodal noise suppression. This method controls a microphone to collect voice data in a dialogue scenario and controls an image acquisition device to capture lip movement video of the dialogue scenario. The method then determines the data matching result based on the voice data, the lip movement video, and a preset large language model. Compared to traditional single-band filtering and denoising algorithms, this method combines data from different modalities, i.e., image data, for denoising processing. The denoising result is determined based on the data matching result and the voice data, thereby avoiding voice distortion or information loss caused by the overlap of user voice and background noise frequency bands during user speech, thus improving the accuracy of noise reduction. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of the structure of a voice interaction device based on multimodal noise suppression in the hardware operating environment of the embodiment of the present invention; Figure 2This is a flowchart illustrating the first embodiment of the voice interaction method based on multimodal noise suppression according to the present invention. Figure 3 This is a flowchart illustrating a second embodiment of the voice interaction method based on multimodal noise suppression according to the present invention. The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0016] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0017] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a voice interaction device based on multimodal noise suppression in the hardware operating environment of the embodiment of the present invention.
[0018] like Figure 1 As shown, the voice interaction device based on multimodal noise suppression may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, an interaction device 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The interaction device 1003 may include a display screen and an input unit such as a keyboard. Optionally, the interaction device 1003 may also be connected to the communication bus via a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0019] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on voice interaction devices based on multimodal noise suppression, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0020] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and a voice interaction program based on multimodal noise suppression.
[0021] exist Figure 1 In the voice interaction device based on multimodal noise suppression shown, the network interface 1004 is mainly used for data communication with other devices; the interaction device 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the voice interaction device based on multimodal noise suppression of the present invention can be set in the voice interaction device based on multimodal noise suppression. The voice interaction device based on multimodal noise suppression calls the voice interaction program based on multimodal noise suppression stored in the memory 1005 through the processor 1001 and executes the voice interaction method based on multimodal noise suppression provided in the embodiment of the present invention.
[0022] This invention provides a voice interaction method based on multimodal noise suppression, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a voice interaction method based on multimodal noise suppression according to the present invention.
[0023] In this embodiment, the voice interaction method based on multimodal noise suppression includes: Step S1: Control the microphone to collect voice data in the dialogue scene, and control the image acquisition device to capture video of lip movements in the dialogue scene; In this embodiment, the microphone and the image acquisition device can be located in the same device. Typically, the camera simultaneously captures the voice data and lip movement video. In practice, the camera usually captures the entire dialogue scene; that is, after obtaining the original image data, the lip movement video is obtained by cropping the original image data. The number of lip movement videos obtained after cropping can be one or more. Optionally, in some scenarios, there are multiple image acquisition devices, each acquiring a corresponding lip movement video.
[0024] Step S2: Determine the data matching result based on the speech data, the lip movement video, and the preset large language model. The data matching result includes: time matching result and sound source matching result. In this embodiment, target prompt words are determined based on the current dialogue scenario. The dialogue scenario is generally preset by the user or determined based on the overall image of the dialogue scenario captured by a camera; common scenarios include construction sites and outdoor scenes during typhoons. Specifically, a mapping table is typically used to determine the target prompt words. The target prompt words can generally include environmental markers, noise types, and the number of sound sources. Of course, the above is not a limitation on the target prompt words. Preferably, since the model needs to output in a standardized format, the target prompt words may also include limiting words that restrict the output format. The speech data, the lip movement video, and the target prompt words are input into the preset large language model to determine the data matching result. In this embodiment, time matching refers to aligning the extracted features from the speech data and the lip movement video in time. Based on the lip movement video, the source of the sound output is identified from the extracted features from the speech data.
[0025] Step S3: Determine the denoising result based on the data matching result and the speech data.
[0026] In this embodiment, optionally, based on the data matching result, the frequency band of the main speaker is marked, and environmental noise from the non-matching direction is adaptively filtered and suppressed. For the time period when the main speaker is speaking, but the moment is not clearly recorded in the voice data, it is supplemented by the pronunciation corresponding to the lip movement video, thereby obtaining the denoising result.
[0027] In this embodiment, a microphone is controlled to collect voice data in a dialogue scenario, and an image acquisition device is controlled to capture lip movement video in the dialogue scenario. The data matching result is determined based on the voice data, the lip movement video, and a preset large language model. Compared with traditional single-band filtering and denoising algorithms, this method can combine data from different modalities, i.e., image data, for denoising processing. The denoising result is determined based on the data matching result and the voice data, thereby avoiding voice distortion or information loss caused by the overlap of user voice and background noise frequency bands during user speech, thus improving the accuracy of noise reduction.
[0028] Furthermore, based on the first embodiment, a second embodiment of the voice interaction method based on multimodal noise suppression of the present invention is proposed. In this embodiment, reference is made to... Figure 3 The step of determining the data matching result based on the speech data, the lip movement video, and the preset large language model includes: Step S21: Input the speech data and the lip movement video into the preset large language model for feature extraction to obtain speech feature vector and visual feature vector; In this embodiment, the speech feature vector includes: recording timestamps, phoneme data, intonation fluctuation data, voiceprint feature data, and dense data of environmental noise distribution. The phoneme data is based on the premise that speech is the smallest unit of speech defined artificially; one articulation action corresponds to one phoneme, and sounds produced by the same articulation action are considered the same phoneme, while different articulation actions correspond to different phonemes. Phonemes are generally classified based on the natural attributes of speech. The visual feature vector includes: recording timestamps, mouth opening and closing data, mouth opening and closing speed, and the visibility of teeth and tongue.
[0029] Step S22: Determine the time matching result based on the feature distribution patterns of the speech feature vector and the visual feature vector; Optionally, the first utterance time of the speaker, detected by visual monitoring, is determined based on the visual feature vector, and a corresponding first utterance marker is generated. The speech feature vector is then marked based on the first utterance time and the recording timestamp of the speech feature vector. When the lip movement video involves multiple users, the number of corresponding visual feature vectors is also multiple, and the number of corresponding first utterance markers is also multiple. However, different first utterance markers correspond to different users, thus marking multiple users' markers in the speech feature vector. It should be explained that the marking here is based on the visual feature vector, which is different from other methods that directly use pure sound. It is equivalent to providing a standard sound marking method in a complex noisy environment. Optionally, the time matching result can be: the marked speech feature vector. Optionally, in other embodiments, the time matching result can also be: a set of deviation vectors that can be used to align the speech feature vector and the visual feature vector.
[0030] Step S23: Determine the sound source matching result based on the lip state information in the visual feature vector and the phoneme data in the speech feature vector.
[0031] In this embodiment, lip state information refers to information related to lip state in the visual feature vector, such as timestamps, which are not related to lip state. In this embodiment, the phoneme category is determined based on the lip state information and matched with the phoneme data of the speech data. Typically, after a successful match, targeted filtering is performed on the Mel-spectrum and formant trajectories corresponding to the phoneme category of the speaker during the first utterance time, thereby obtaining accurate denoising results.
[0032] In this embodiment, the speech data and the lip movement video are input into the preset large language model for feature extraction to obtain speech feature vectors and visual feature vectors. Based on the feature distribution patterns of the speech feature vectors and the visual feature vectors, the time matching result is determined. Based on the lip state information in the visual feature vectors and the phoneme data in the speech feature vectors, the sound source matching result is determined. This allows the use of data from multiple modalities for sound source matching, thereby improving the accuracy of subsequent filtering and noise reduction.
[0033] Furthermore, based on the first or second embodiment, a third embodiment of the speech interaction method based on multimodal noise suppression of the present invention is proposed. In this embodiment, the step of determining the denoising result based on the data matching result and the speech data includes: Based on the time matching result, the speech data and the lip movement video are time-stamp aligned to obtain aligned multimodal data; Based on the sound source matching results, the target speech stream of the target speaker is located from the aligned multimodal data; The target speech stream is subjected to background noise suppression processing to obtain the denoising result.
[0034] In this embodiment, the time matching result is a deviation vector used to align the speech feature vector and the visual feature vector. Based on the deviation vector, the speech data and the lip movement video are time-stamp aligned, so that the speech data can be identified by the timestamp as a corresponding first phonation marker. In this embodiment, the sound source matching result provides the speaker's phoneme category, thereby enabling the determination of the target speech stream based on the sound source matching result.
[0035] Furthermore, the step of performing background noise suppression processing on the target speech stream to obtain the denoising result includes: Lip shape feature data are determined based on the lip movement video; When the signal-to-noise ratio of the target speech stream is lower than a preset threshold, the missing phonemes in the target speech stream are inferred based on the lip shape feature data to obtain an enhanced speech stream. The enhanced speech stream is used as the denoising result.
[0036] In this embodiment, when strong environmental noise causes data interference of some phonemes in the target speech stream, lip shape feature data is used as a parallel clue; the mouth opening and closing and tongue and teeth visibility captured by the visual channel can provide prior knowledge of the pronunciation position and mode during periods of weak acoustic signal, thereby filling in the spectral details erased by noise, realizing visual-assisted speech enhancement, and making the output speech sound continuous and natural.
[0037] Further, the step of performing auxiliary inference on the missing phonemes in the target speech stream according to the lip feature data to obtain an enhanced speech stream includes: Determine a visual phoneme sequence according to the lip feature data; Calculate the association probability distribution between the visual phoneme sequence and the remaining audio features in the target speech stream; Determine the predicted compensation features of the missing phonemes according to the association probability distribution; Perform signal reconstruction processing on the target speech stream according to the predicted compensation features to obtain the enhanced speech stream.
[0038] In this embodiment, the visual phoneme sequence here is determined by one or more lip movements. Specifically, in the process of calculating the association probability distribution between the visual phoneme sequence and the remaining audio features in the target speech stream, a phoneme association correspondence table can be preset, and the predicted compensation features of the missing phonemes are determined according to the phoneme association correspondence table and the visual phoneme sequence. Specifically, the defective section of the target speech stream is inserted frame by frame to align, so that the spectrogram is restored to be continuous, thereby obtaining the enhanced speech stream.
[0039] Further, based on any of the above embodiments, a fourth embodiment of the speech interaction method based on multi-modal noise suppression according to the present invention is proposed. In this embodiment, after the step of determining the denoising result according to the data matching result and the speech data, the following steps are further included: Determine the recognized text according to the denoising result; Determine the target text result according to the recognized text and a preset dialect adaptation module; Among them, the step of determining the target text result according to the recognized text and the preset dialect adaptation module includes: Determine the fine-tuned model parameters according to the recognized text and a small number of dialect samples through the Few-shot learning strategy, and determine the target text result according to the fine-tuned model parameters.
[0040] In this embodiment, it should be noted that by using Few-shot, the recognition error in the dialect scenario In this embodiment, Few-shot dialect adaptation: By fine-tuning the model through transfer learning, only 50 dialect speech samples are required to adjust the phoneme feature weights. For example, tests on Cantonese users show that the recognition error rate of the acoustic model for "冇" and Cantonese "没有" has dropped from 18% to 9%, and the key instruction understanding accuracy has been increased to 92% from 75%.
[0041] Further, after the step of determining the target text result according to the recognized text and the preset dialect adaptation module, the following steps are further included: Determine context information based on the target text result and the conversation history; Based on the context information, determine the user's intention through the semantic reasoning ability of the preset large language model.
[0042] In addition, for enhancing context association: Combining the semantic reasoning ability of the large language model, the system can infer the meaning of dialect words based on the conversation history. For example, the Sichuan-Chongqing dialect word "晓得" corresponds to "知道" (know), reducing misjudgments caused by regional expression differences.
[0043] In addition, an embodiment of the present invention further provides a voice interaction device based on multimodal noise suppression, which is characterized in that the device includes: A control module, configured to control the microphone to collect voice data in the conversation scene and control the image acquisition device to capture the lip movement video in the conversation scene; An alignment module, configured to determine a data matching result based on the voice data, the lip movement video, and a preset large language model, and the data matching result includes: a time matching result and a sound source matching result; A denoising module, configured to determine the denoising result based on the data matching result and the voice data.
[0044] In addition, an embodiment of the present invention further provides a voice interaction device based on multimodal noise suppression. The voice interaction device based on multimodal noise suppression includes: a memory, a processor, and a voice interaction program based on multimodal noise suppression stored on the memory and executable on the processor. The voice interaction program based on multimodal noise suppression is configured to implement the steps of the voice interaction method based on multimodal noise suppression described in any one of the above.
[0045] In addition, an embodiment of the present invention further provides a storage medium. A voice interaction program based on multimodal noise suppression is stored on the storage medium. When the voice interaction program based on multimodal noise suppression is executed by a processor, it implements the steps of the voice interaction method based on multimodal noise suppression described in any one of claims 1 to 7.
[0046] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0047] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0048] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0049] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A voice interaction method based on multimodal noise suppression, characterized in that, The voice interaction method based on multimodal noise suppression includes the following steps: The microphone is controlled to collect voice data in the dialogue scene, and the image acquisition device is controlled to capture video of lip movements in the dialogue scene; The data matching results are determined based on the speech data, the lip movement video, and the preset large language model. The data matching results include: time matching results and sound source matching results. The denoising result is determined based on the data matching result and the speech data.
2. The voice interaction method based on multimodal noise suppression as described in claim 1, characterized in that, The step of determining the data matching result based on the speech data, the lip movement video, and the preset large language model includes: The speech data and the lip movement video are input into the preset large language model for feature extraction to obtain speech feature vectors and visual feature vectors. The time matching result is determined based on the feature distribution patterns of the speech feature vector and the visual feature vector; The sound source matching result is determined based on the lip state information in the visual feature vector and the phoneme data in the speech feature vector.
3. The voice interaction method based on multimodal noise suppression as described in claim 1, characterized in that, The step of determining the denoising result based on the data matching result and the speech data includes: Based on the time matching result, the speech data and the lip movement video are time-stamp aligned to obtain aligned multimodal data; Based on the sound source matching results, the target speech stream of the target speaker is located from the aligned multimodal data; The target speech stream is subjected to background noise suppression processing to obtain the denoising result.
4. The voice interaction method based on multimodal noise suppression as described in claim 3, characterized in that, The step of performing background noise suppression processing on the target speech stream to obtain the denoising result includes: Lip shape feature data are determined based on the lip movement video; When the signal-to-noise ratio of the target speech stream is lower than a preset threshold, the missing phonemes in the target speech stream are inferred based on the lip shape feature data to obtain an enhanced speech stream. The enhanced speech stream is used as the denoising result.
5. The voice interaction method based on multimodal noise suppression as described in claim 4, characterized in that, The step of performing auxiliary inference on missing phonemes in the target speech stream based on the lip shape feature data to obtain the enhanced speech stream includes: The visual phoneme sequence is determined based on the lip shape feature data; Calculate the probability distribution of the association between the visual phoneme sequence and the residual audio features in the target speech stream; The predictive compensation features for missing phonemes are determined based on the aforementioned correlation probability distribution; The target speech stream is reconstructed based on the predicted compensation features to obtain the enhanced speech stream.
6. The voice interaction method based on multimodal noise suppression as described in any one of claims 1 to 5, characterized in that, After the step of determining the denoising result based on the data matching result and the speech data, the method further includes: The text to be recognized is determined based on the denoising results; The target text result is determined based on the identified text and the preset dialect adaptive module; The step of determining the target text result based on the recognized text and the preset dialect adaptation module includes: Based on the identified text and a small number of dialect samples, the fine-tuned model parameters are determined using a Few-shot learning strategy, and the target text result is determined based on the fine-tuned model parameters.
7. The voice interaction method based on multimodal noise suppression as described in claim 6, characterized in that, After the step of determining the target text result based on the recognized text and the preset dialect adaptive module, the method further includes: Contextual information is determined based on the target text result and the dialogue history; Based on the context information, the user's intent is determined through the semantic reasoning capabilities of the preset large language model.
8. A voice interaction device based on multimodal noise suppression, characterized in that, The device includes: The control module is used to control the microphone to collect voice data in the dialogue scenario and to control the image acquisition device to capture video of lip movements in the dialogue scenario. The alignment module is used to determine the data matching result based on the speech data, the lip movement video, and the preset large language model. The data matching result includes: time matching result and sound source matching result. A noise reduction module is used to determine the noise reduction result based on the data matching result and the speech data.
9. A voice interaction device based on multimodal noise suppression, characterized in that, The voice interaction device based on multimodal noise suppression includes: a memory, a processor, and a voice interaction program based on multimodal noise suppression stored in the memory and executable on the processor, wherein the voice interaction program based on multimodal noise suppression is configured to implement the steps of the voice interaction method based on multimodal noise suppression as described in any one of claims 1 to 7.
10. A storage medium, characterized in that, The storage medium stores a voice interaction program based on multimodal noise suppression, which, when executed by a processor, implements the steps of the voice interaction method based on multimodal noise suppression as described in any one of claims 1 to 7.