Multi-modal classification method and system for audio semantic understanding and rejection

Abstract

The embodiment of the invention provides a multi-modal classification method and system for audio semantic understanding and rejection. The method comprises the following steps: receiving a collected original audio signal; acquiring a context historical memory of the original audio signal in voice interaction, and capturing multi-dimensional features related to semantic understanding and rejection from the original audio signal and the context historical memory; inputting the multi-dimensional features into a pre-trained multi-modal classification large model, wherein the multi-modal classification large model directly determines speech recognition content, semantic classification and rejection results of the original audio signals by using the multi-dimensional features; and determining a reply result for feeding back the original audio signal based on the speech recognition content, the semantic classification and the rejection result. According to the embodiment of the invention, end-to-end processing from audio to semantic understanding and rejection is realized. The speech recognition capability and the speech classification and rejection capability are unified in the multi-modal large model, so that the calculation resources are reduced, the overall system delay is greatly reduced, and the speech recognition accuracy is also improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent voice, in particular to a multi-modal classification method and system for audio semantic understanding and rejection. BACKGROUND

[0002] In a voice dialogue, the ASR (Automatic Speech Recognition) system is combined with a text classification model to perform semantic understanding and voice rejection based on confidence to realize interactive dialogue with the user.

[0003] In the prior art, the following two technologies are combined to perform voice dialogue: the ASR system is combined with a text classification model to perform semantic understanding: this is the current mainstream voice combined semantic understanding scheme, which first converts the voice signal into text, and then classifies the text to realize semantic judgment; voice rejection based on confidence: in the ASR system, by setting a recognition confidence threshold, the recognition result with a confidence lower than the threshold is rejected.

[0004] In the process of implementing the present application, the inventors found that the related art at least has the following problems: 1. Recognition error propagation problem: the existing technology uses a serial processing mode for the ASR system to convert audio into text, which causes the recognition error of the ASR to directly affect the subsequent text classification result. For example, in a noisy environment, at a fast speech speed, with a strong accent or dialect, the recognition accuracy of the ASR will decrease, and the resulting error text will directly cause the semantic understanding to deviate, and even result in a completely wrong classification result, that is, the ASR as a front-end module, its recognition accuracy directly determines the upper limit of the subsequent text classification.

[0005] 2. Cannot directly make deep understanding based on audio features, especially reject "meaningless voice" or "non-human speech", and the rejection ability for non-human sounds and background noise is insufficient: the existing ASR system is mainly designed for human voice, and for animal sounds, machine roar, environmental noise and other non-human sounds, the ASR cannot convert them into meaningful text, but the system may still attempt to process, resulting in invalid or incorrect response. Even the rejection based on confidence is often performed after the ASR attempts to recognize, and cannot directly judge and reject these non-voice signals at a lower level, that is, cannot directly make deep understanding based on audio features.

[0006] 3. Robustness is insufficient: the existing system is poor in adaptability to environmental conditions (such as noise, reverberation), speaker characteristics (such as accent, speech rate, volume). In complex and variable real application scenarios, the performance of the system is often difficult to achieve the ideal state, and misrecognition or missed recognition is easy to occur. Even if the ASR confidence is combined to jointly classify, the influence of non-speech information can be reduced with high probability, but it still depends on the ability of the ASR model itself, and the robustness of this cascading scheme is low, which is a long-standing problem in this field. SUMMARY

[0007] In order to at least solve the problems in the prior art that ASR affects semantic understanding, cannot directly make deep understanding based on audio features, and has low robustness.

[0008] In a first aspect, an embodiment of the present application provides a multi-modal classification method for audio semantic understanding and rejection, comprising: receiving a collected original audio signal, wherein the original audio signal includes user dialogue sound, non-human vocalization, and environmental noise; obtaining a context historical memory of the original audio signal in voice interaction, capturing multi-dimensional features related to semantic understanding and rejection from the original audio signal and the context historical memory, wherein the context historical memory includes input audio of a historical dialogue of voice interaction of the original audio signal, and a reply result of feedback of the input audio; inputting the multi-dimensional features into a pre-trained multi-modal classification large model, the multi-modal classification large model directly determines voice recognition content, semantic classification, and rejection result of the original audio signal using the multi-dimensional features, wherein the multi-modal classification large model is an end-to-end architecture; determining a reply result of feedback of the original audio signal based on the voice recognition content, semantic classification, and rejection result.

[0009] In a second aspect, an embodiment of the present application provides a multi-modal classification system for audio semantic understanding and rejection, comprising: a collection module configured to receive a collected original audio signal, wherein the original audio signal includes user dialogue sound, non-human vocalization, and environmental noise; a feature extraction module configured to obtain a context historical memory of the original audio signal in voice interaction, and capture multi-dimensional features related to semantic understanding and rejection from the original audio signal and the context historical memory, wherein the context historical memory includes input audio of a historical dialogue of voice interaction of the original audio signal, and a reply result of feedback of the input audio; a multi-modal classification module, configured to input the multi-dimensional features into a pre-trained multi-modal classification large model, and the multi-modal classification large model is configured to determine speech recognition content, semantic classification and rejection result of the original audio signal directly by using the multi-dimensional features, wherein the multi-modal classification large model is an end-to-end architecture; a result determination module, configured to determine a reply result of the original audio signal based on the speech recognition content, the semantic classification and the rejection result.

[0010] In a third aspect, an electronic device is provided, which includes at least one processor and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the multi-modal classification method for audio semantic understanding and rejection according to any one of the embodiments of the present application.

[0011] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the steps of the multi-modal classification method for audio semantic understanding and rejection according to any one of the embodiments of the present application.

[0012] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, wherein the computer program / instruction is executed by a processor to implement the steps of the multi-modal classification method for audio semantic understanding and rejection according to any one of the embodiments of the present application.

[0013] The multi-modal classification large model with an end-to-end architecture of the method discards the traditional cascade mode of "ASR+text classification", directly inputs the original audio signal into a unified multi-modal large model, and realizes end-to-end processing from audio to semantic understanding and rejection. Furthermore, the speech recognition capability and the speech classification and rejection capability are unified in one multi-modal large model, which reduces the computing resources, greatly reduces the overall system delay, and improves the accuracy of speech recognition. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0015] Figure 1 is a flowchart of a multi-modal classification method for audio semantic understanding and rejection provided by an embodiment of the present application; Figure 2 is a whole flow schematic diagram of a multi-modal classification method for audio semantic understanding and recognition provided by an embodiment of the present application; Figure 3 is a structural schematic diagram of a multi-modal classification system for audio semantic understanding and recognition provided by an embodiment of the present application; Figure 4 is a structural schematic diagram of an embodiment of an electronic device for multi-modal classification of audio semantic understanding and recognition provided by an embodiment of the present application. DETAILED DESCRIPTION

[0016] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0017] As Figure 1 shown is a flowchart of a multi-modal classification method for audio semantic understanding and recognition provided by an embodiment of the present application, including the following steps: S11: receiving a collected original audio signal, wherein the original audio signal includes user dialogue sound, non-human vocalization and environmental noise; S12: obtaining a context history memory in voice interaction of the original audio signal, capturing multi-dimensional features related to semantic understanding and recognition from the original audio signal and the context history memory, wherein the context history memory includes input audio of a voice interaction history dialogue of the original audio signal, and a reply result of feedback of the input audio; S13: inputting the multi-dimensional features into a pre-trained multi-modal classification large model, the multi-modal classification large model directly determines voice recognition content, semantic classification and recognition result of the original audio signal by using the multi-dimensional features, wherein the multi-modal classification large model is an end-to-end architecture; S14: determining a reply result of feedback of the original audio signal based on the voice recognition content, semantic classification and recognition result.

[0018] In this embodiment, the core idea of the multi-modal classification method for audio semantic understanding and recognition is "audio implantation of large model", that is, the traditional ASR text conversion link is skipped, and the original audio signal is directly taken as input, and a multi-modal classification large model for audio semantic understanding and recognition is used to complete the semantic understanding and recognition (recognition refers to "meaningless speech" or "non-human speech" without converting it into text) or classification task, and multi-round context memory is integrated for more accurate judgment.

[0019] However, the above idea is difficult to implement: the existing voice recognition forms a fixed way of thinking of "first converting speech into text, and then corresponding processing", and the present method changes this way to overcome the prejudice of the skilled person in the art for this fixed way of processing speech. In addition, directly processing the original audio signal and extracting high-level semantic information from it puts very high requirements on the architecture and training of the model. The original audio is continuous, high-dimensional and has a huge amount of information. How to design a large model that can effectively capture its time sequence characteristics, acoustic characteristics and deep patterns associated with semantics is a great technical challenge. Since the present method trains a large model that can directly understand semantics from audio and perform recognition, a large amount of "audio-semantic label" data is required, and these data need to contain rich "meaningless speech" and "non-human speech" samples. Compared with "audio-text" data, such data is more difficult to obtain and label, and is more difficult to obtain.

[0020] That is, the present method constructs an end-to-end multi-modal intelligent classification large model, which can directly receive the original audio signal as input and output semantic classification results or recognition judgment, and output the ASR speech recognition result as the input of the downstream task, and can integrate multi-round context history memory for more accurate decision-making. It has the following functions: 1. Multi-modal feature extraction: the model no longer relies only on the text features output by ASR, but extracts multi-dimensional features from the original audio.

[0021] 2. Unified large model processing and context fusion: the extracted multi-modal features and multi-round context history memory (including the input audio of the historical dialogue and the output result text of the historical dialogue) are input into the multi-modal large model. It can understand multi-round input content, judge whether the latest input speech is in dialogue with the intelligent assistant, and make accurate classification judgment and output speech recognition content. For example, when the user is talking with other passengers in the car, even if the speech content is clear, the model can judge according to the context that it is not an instruction to the intelligent assistant, and thus perform recognition.

[0022] For step S11, the method can be applied to various voice devices, such as smart speakers and in-vehicle central control head units. These devices collect raw audio signals in real time from the current environment, which may be background noise in the environment, such as the background noise of a washing machine running in a home scenario, or engine noise in a vehicle scenario, non-human sounds including AI electronic sounds, and user dialogue sounds, such as user voice input to the device and user dialogue with other users.

[0023] For step S12, after obtaining the raw audio, if there is a corresponding context history memory in the raw voice, it should be noted that the context history memory used by the method includes the input audio of the voice interaction history dialogue and the reply result of the smart device carrying the method in response to the input audio. It is not the dialogue and reply between two speakers.

[0024] Capture multi-dimensional features related to semantic understanding and recognition, including voice content, prosodic features reflecting the speaker's expression and emotion, accent, speed, environmental acoustic features for understanding audio content and determining recognition, noise and reverberation, which help determine the recognition of the input voice or the semantic understanding of the corresponding scene.

[0025] For step S13, the multi-dimensional features determined in step S12 are input into a pre-trained multi-modal classification large model, and the multi-modal classification large model of the method directly determines the voice recognition content, semantic classification and recognition result of the raw audio signal from the multi-dimensional features. For example, the content of the collected raw audio signal is "navigate to KFC", which is input into the end-to-end multi-modal classification large model to directly determine the multi-modal classification result "semantic category: navigation; voice recognition content: navigate to KFC, no recognition"; the content of the collected raw audio signal is "user's exclamation in a noisy vehicle environment: what's wrong with the car in front", which is input into the end-to-end multi-modal classification large model to directly determine the multi-modal classification result "semantic category: none; voice recognition content: what's wrong with the car in front, recognition"; the content of the collected raw audio signal is "pure environmental noise: zizi... watermelon... zizi... yuan", which is input into the end-to-end multi-modal classification large model to directly determine the multi-modal classification result "semantic category: none; voice recognition content: none, recognition". The semantic category can also include: car control, music, navigation, news, etc.

[0026] For step S14, the reply result is determined based on the voice recognition content, semantic classification and recognition result described above, for example, in the above three examples, only "navigate to KFC" is replied. Other examples are not the content input by the user to the smart device, and no reply is made for recognition.

[0027] The above multi-modal classification for audio semantic understanding and recognition is based on an end-to-end multi-modal classification large model implemented by the method, and the end-to-end architecture multi-modal classification large model comprises: An audio input module configured to acquire original audio signals including user dialogue, non-human vocalization, and environmental noise; A multi-modal feature extraction module configured to capture multi-dimensional features of the original audio signals and context history, wherein the multi-dimensional features include speech content, prosodic features reflecting speaker expression and emotion, and environmental acoustic features for understanding audio content and determining recognition; A multi-modal large model processing module comprising a semantic classification and recognition judgment module to establish a feature structure relationship between multi-modal features and speech recognition content, semantic classification, and recognition results, configured to receive the multi-dimensional features and directly determine the speech recognition content, semantic classification, and recognition results of the original audio signals; A result output module configured to output a reply result of the speech recognition content, semantic classification, and recognition results.

[0028] The training of the end-to-end architecture multi-modal classification large model comprises: An audio input module configured to acquire training audio including user dialogue, non-human vocalization, and environmental noise, context history related to the training audio, and benchmark multi-modal classification labels; A multi-modal feature extraction module configured to determine multi-modal features of the training audio; A multi-modal large model processing module configured to determine predicted multi-modal classification labels including speech recognition content, semantic classification, and recognition results from the multi-modal features; The benchmark multi-modal classification labels are used as a guide to determine the error between the predicted multi-modal classification labels and the benchmark multi-modal classification labels, and the end-to-end architecture multi-modal classification large model is trained based on the error until a preset target is reached.

[0029] In this embodiment, in order to train the multi-modal large model of the method, original audio signals are received from a microphone or other audio acquisition device. The audio signals can be user dialogue, instructions, or background noise, non-human vocalization, etc. in the environment. In a vehicle-mounted scenario, this is usually user speech and environmental sound collected by an in-vehicle microphone array.

[0030] More specifically, in order to give the model enough data for training, and to achieve semantic understanding and rejection, the method uses a device with a microphone to obtain directly collected original audio signals. For example, a smart speaker, which is placed in different scenarios (for example, a home scenario, or a vehicle scenario). In these scenarios, not only the user's interactive voice in the scene is collected, but also the environmental sound when there is no user voice, including: background environmental noise in the home scenario, such as the environmental noise of the washing machine running, or the background environmental noise of the robot sweeping, or the environmental noise of the range hood when cooking. Vehicle environmental noise in the vehicle scenario, such as environmental noise when the vehicle is running, environmental noise when the vehicle is stopped but the engine is running, etc. Through the above examples, that is, the method collects background noise in various environments, which will not be repeated here. For the voice with the user or the speaker, the user's voice can be directly recorded, or the user's interactive voice with the smart device can be collected with the user's consent. After the conversation ends, the user's voice is automatically collected. For example, interactive voice for controlling the smart TV with the environmental noise of the washing machine running, or navigation interactive voice in the vehicle noise environment.

[0031] After obtaining the original audio, if there is context in the original voice, for example, in the multi-round conversation between the user and the smart device, there will be context. At this time, when there is context, the text content of the context is additionally collected, for example, through voice-to-text. The context voice of the original audio and the text are used as the context history memory of the original audio. Through the above steps, the data for training the model is obtained. (It should be noted that the voice-to-text technology in data processing is not used in the model of the method).

[0032] For training data of pure noise and non-human voice, according to the characteristics of voice fluctuation, timbre, etc., the reference multi-modal classification label can be directly marked as a rejection classification label, and for multi-round conversation training between the user and the smart device, the ability to continue the conversation indicates that the reference multi-modal classification label is not rejection, and through the content of voice-to-text, the corresponding specific semantic classification label is determined. Through the above steps, the training audio including user conversation voice, non-human voice and environmental noise, the context history memory related to the training audio and the reference multi-modal classification label are obtained. For the acquisition of training data, targeted acquisition can also be performed according to the needs, which will not be repeated here.

[0033] After obtaining the training audio, further multi-modal feature extraction is performed. Multi-dimensional feature extraction is performed on the training audio (which can also be referred to as the original training audio). If the training audio has corresponding context historical memory, multi-dimensional feature extraction is performed on the corresponding context voice and text. It should be noted that this step aims to comprehensively capture various information related to semantic understanding and recognition from the original training audio, rather than just the voice content. The different types of features related to semantic understanding and recognition described above are integrated to form a rich multi-modal feature vector, which is used as the input of the subsequent multi-modal large model processing module.

[0034] The multi-dimensional features can use a sound encoder to encode the input audio to obtain hidden feature representations such as sound content, prosodic features, and environmental acoustic features. The prosodic features include, for example, speech rate, pauses, stress, and the like, reflecting the speaker's expression and emotion. The environmental acoustic features include capturing the acoustic patterns of background noise and non-human vocalizations to better understand the audio content and make more accurate recognition or classification.

[0035] To further improve the classification effect of the multi-modal large model, the context historical memory of multi-round dialogue can also be integrated to achieve more refined recognition. By incorporating context historical memory in training, the trained multi-modal large model can remember and utilize the multi-round dialogue context to intelligently determine whether the user is interacting with the intelligent assistant. For example, when the user is talking to other people in the car, the system can recognize and reject (determine that, even though there is dialogue interaction, the object of interaction is actually other people, not the intelligent device itself), avoiding false touch and unnecessary interruption, greatly improving the naturalness of the interaction and user experience.

[0036] The multi-modal feature vector obtained by the above steps is input into the processing module of the multi-modal large model that best embodies the function of the method. To fundamentally solve the problem of recognition error propagation, the multi-modal large model of the method skips the ASR text conversion link and directly performs semantic understanding and recognition from the original audio / multi-modal features, reducing the error accumulation of cascaded models and avoiding the influence of ASR errors on subsequent tasks, thereby improving the overall accuracy and robustness of the system.

[0037] The multi-modal large model of the method is an end-to-end deep learning model, including an audio input module, a multi-modal feature extraction module, a multi-modal large model processing module, a refined recognition judgment module, a semantic classification module, and a result output module.

[0038] The original audio and the corresponding context history memory (audio + text) are received simultaneously through the audio input module. Through training of large-scale audio data and the corresponding context history memory and the pre-prepared benchmark multi-modal classification label, multi-dimensional audio features and multi-turn context are deeply fused and semantically understood. The end-to-end large model architecture adopts the core processing flow of the "audio direct multi-modal large model", which fundamentally solves the problem of recognition error propagation.

[0039] Specifically, the training data obtained in the above step includes original audio, context history memory, multi-modal features, and pre-prepared benchmark multi-modal classification labels (speech of a certain scene, semantic understanding result, and rejection result) corresponding to each original audio.

[0040] In the training, the training data is input into the multi-modal classification large model of the end-to-end architecture, the multi-modal feature extraction module is used to determine the multi-modal features of the training data, and the fine rejection judgment module and the semantic classification module of the multi-modal large model processing module are used to comprehensively determine the predicted semantic classification, the predicted speech recognition text, and the rejection result. In the multi-modal large model, the feature structure relationship between the original audio, the context history memory, the multi-modal features, and the recognition result is established through each module; and the multi-modal features and the benchmark multi-modal classification label are used as a guide, so that the multi-modal large model can directly extract the predicted semantic and the predicted speech recognition text from the original audio and the context history memory through the feature structure relationship, and directly correspond to the predicted recognition result.

[0041] The multi-modal large model is trained in reverse propagation using the error between the predicted recognition result and the benchmark recognition result until the error reaches the preset target.

[0042] As an implementation, for the training of the model, there are different training methods. The overall model is trained through the error, and the trained multi-modal classification large model directly receives the audio signal and outputs the multi-modal classification result.

[0043] If it is to train the multi-modal classification core module more accurately, the training includes freezing the audio input module, the multi-modal feature extraction module, and the result output module, and only training the parameters in the multi-modal large model processing module. In this way, the multi-modal large model processing module can be directly used as a pre-trained multi-modal classification large model after training, and the audio signal acquisition, multi-dimensional features, and reply and other module processing. In this way, the multi-modal classification large model can be more simplified and modularized, the complexity of system design, development, and maintenance is simplified, and the overall latency is lower.

[0044] As an implementation, the speech recognition content, semantic classification, and rejection result of the original audio signal, and the reply result of the original audio signal are used to update the context history memory.

[0045] In addition to the above training, the context history memory is updated by the predicted semantics and the predicted speech recognition text, and the reply content in the subsequent dialogue, The context history memory includes context speech and text. The predicted semantics and the predicted speech recognition text, and the reply content in the subsequent dialogue, which are identified by the multi-modal large model in this multi-modal model recognition, also belong to the context content required by the method. The context history memory is updated by the predicted semantics and the predicted speech recognition text, and the reply content in the subsequent dialogue. The update of the context history memory also improves the training effect of the multi-modal large model.

[0046] For example, in this round of dialogue, the multi-modal large model identifies that the user input is “open the rear left window”, and the subsequent intelligent voice replies “the window is opened”. The context history memory is updated by these speech and text. Similarly, the corresponding update will also be performed for the rejection scene. For example, the user input “what movie did you say yesterday” is collected (actually the user is talking to the person next to him, not to the intelligent device). At this time, if the intelligent device replies incorrectly, in the next round of dialogue, the user may input “stop, didn't call you, or ignore the intelligent device and continue to talk to the person next to him”. These contents have no direct connection with the incorrect reply of the intelligent device, and the contents are updated to the context history memory. In this way, the multi-modal large model updates the recognition result of this speech from “recognition” to “rejection” during training. Conversely, the user inputs “open the window”, and the multi-modal large model incorrectly rejects it. In the next round of dialogue, the user repeatedly inputs “open the window”. The contents are updated to the context history memory, which indicates that for this speech content, it should not be rejected but should be replied. In this way, in the subsequent training, the error is corrected.

[0047] Through the above training, the multi-modal large model can learn and capture more rich acoustic, prosodic and potential semantic features from the original audio, so as to realize a deeper understanding of the speech content. Compared with the traditional ASR, the multi-modal large model can output more accurate speech recognition results, and at the same time output the speech recognition content, make intent classification or make rejection processing, for example, when the user is in dialogue with the intelligent device, output the semantic classification, the reply corresponding to the speech recognition result, or when the user is not in dialogue with the intelligent device, automatically make a rejection judgment and do not output any content reply. The multi-modal large model realizes deeper semantic understanding of audio, and enhances the user experience.

[0048] Finally, in general, the multi-modal large model using the method interacts / trains with the user for semantic understanding and classification rejection as Figure 2 shown.

[0049] If it is further training, the user voice is inputted to the multi-modal large model after voice feature extraction. The original audio, context history memory and multi-modal features are inputted to the multi-modal large model. The model training has been described above, and will not be repeated here. If it is the use of the trained multi-modal large model, the user voice is directly inputted to the multi-modal large model, that is, the end-to-end model architecture of the method. The core processing flow of "audio directly inputting the multi-modal large model" is adopted. The multi-modal large model directly outputs semantic classification and speech recognition. The context memory is updated with these contents. The classification results and subsequent interactive information (including speech recognition results, downstream task completion responses) or rejection results of the model are updated to the context memory module for continuous use in subsequent interactions. Then, the classification results of vehicle control, music, navigation, news and rejection are outputted. If it is determined that the user voice needs to be replied, the classification results are outputted. If it is determined to be rejected, no reaction is made. The rejection judgment and semantic classification are based on the direct processing of the multi-modal large model on the original audio features and multi-round context memory, rather than relying on the speech-to-text results. The semantic classification results and speech recognition results are outputted to the subsequent application module or user interface together. These results can be used to drive intelligent assistant responses, execute specific tasks, or perform data analysis, etc.

[0050] As can be seen from the above, the method integrates multiple cascaded modules (text classification, rejection module, context management) into a unified end-to-end multi-modal large model, reduces the interface complexity between modules, reduces the difficulty of system integration and debugging, improves the development efficiency and the convenience of later maintenance. The trained end-to-end multi-modal large model can directly receive the original audio signal, realize end-to-end processing from audio to semantic understanding and rejection. It can automatically make rejection judgment for "meaningless voice", "non-human sound", "non-speaking to the system" and other input audio, avoiding the embarrassment of "I didn't understand" or incorrect response. For normal conversation, it can accurately classify and output speech recognition results for downstream tasks. Reducing computing resources reduces the overall system latency and improves the accuracy of speech recognition.

[0051] In general, the multi-modal classification large model of the end-to-end architecture of the method discards the traditional cascade mode of "ASR + text classification", directly inputs the original audio signal into a unified multi-modal large model, and realizes end-to-end processing from audio to semantic understanding and recognition. Further, the speech recognition capability and the speech classification and recognition capability are unified in one multi-modal large model, which reduces the computing resources, greatly reduces the overall system delay, and improves the accuracy of speech recognition. The multi-modal classification system can intelligently judge classification or recognition directly from the acoustic, prosodic, environmental acoustic and potential semantic features of the original audio, combined with multi-round context history memory, realize deeper and more comprehensive semantic understanding of the speech content, and output accurate speech recognition results.

[0052] As shown in Figure 3 The multi-modal classification system for audio semantic understanding and recognition provided by the embodiment of the present application includes the multi-modal classification method for audio semantic understanding and recognition described in any of the above embodiments, and is configured in a terminal.

[0053] The multi-modal classification system for audio semantic understanding and recognition provided by the embodiment of the present application includes the multi-modal classification method for audio semantic understanding and recognition described in any of the above embodiments, and is configured in a terminal.

[0054] The acquisition module 11 is configured to receive the acquired original audio signal, wherein the original audio signal includes user dialogue sound, non-human sound and environmental noise. The feature extraction module 12 is configured to obtain the context history memory in the voice interaction of the original audio signal, and capture multi-dimensional features related to semantic understanding and recognition from the original audio signal and the context history memory, wherein the context history memory includes input audio of the voice interaction history dialogue of the original audio signal, and reply results fed back to the input audio. The multi-modal classification module 13 is configured to input the multi-dimensional features into a pre-trained multi-modal classification large model, and the multi-modal classification large model directly determines the speech recognition content, semantic classification and recognition result of the original audio signal by using the multi-dimensional features, wherein the multi-modal classification large model is an end-to-end architecture. The result determination module 14 is configured to determine the reply results fed back to the original audio signal based on the speech recognition content, semantic classification and recognition result.

[0055] The embodiment of the present application also provides a non-volatile computer storage medium, and the computer storage medium stores computer executable instructions. The computer executable instructions can execute the multi-modal classification method for audio semantic understanding and recognition in any of the above method embodiments. As an implementation manner, the non-volatile computer storage medium of the present application stores computer executable instructions, and the computer executable instructions are configured to: receiving a collected original audio signal, wherein the original audio signal comprises user dialogue, non-human vocalization and environmental noise; obtaining a context history memory of the original audio signal in voice interaction, capturing multi-dimensional features related to semantic understanding and rejection from the original audio signal and the context history memory, wherein the context history memory comprises input audio of a voice interaction history dialogue of the original audio signal, and a reply result of feedback of the input audio; inputting the multi-dimensional features into a pre-trained multi-modal classification large model, the multi-modal classification large model directly determines voice recognition content, semantic classification and rejection result of the original audio signal by using the multi-dimensional features, wherein the multi-modal classification large model is an end-to-end architecture; determining a reply result of feedback of the original audio signal based on the voice recognition content, semantic classification and rejection result.

[0056] As a non-volatile computer readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present application. One or more program instructions stored in the non-volatile computer readable storage medium, when executed by the processor, execute the multi-modal classification method for audio semantic understanding and rejection in any method embodiment described above.

[0057] Figure 4 is the hardware structure schematic diagram of the electronic device for the multi-modal classification method for audio semantic understanding and rejection provided by another embodiment of the present application, as shown in Figure 4 The device comprises: one or more processors 410 and memories 420, Figure 4 Take one processor 410 as an example in the above. The device for the multi-modal classification method for audio semantic understanding and rejection can also comprise input devices 430 and output devices 440.

[0058] The processor 410, the memory 420, the input device 430 and the output device 440 can be connected through a bus or other means, Figure 4 Take the connection through the bus as an example.

[0059] The memory 420, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the multi-modal classification method for audio semantic understanding and rejection provided in the embodiments of the present application. The processor 410 executes various function applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 420, that is, implements the multi-modal classification method for audio semantic understanding and rejection provided in the above method embodiments.

[0060] The memory 420 can include a program area and a data area, where the program area can store an operating system and application programs required by at least one function; and the data area can store data and the like. In addition, the memory 420 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 420 can optionally include a memory remotely arranged with respect to the processor 410, and these remote memories can be connected to the mobile device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0061] The input device 430 can receive input digital or character information. The output device 440 can include a display device such as a display screen.

[0062] The one or more modules are stored in the memory 420 and, when executed by the one or more processors 410, perform the multi-modal classification method for audio semantic understanding and rejection in any of the above method embodiments.

[0063] The above product can perform the method provided in the embodiments of the present application, has the function modules and beneficial effects corresponding to the execution method. Technical details not described in detail in the embodiments can be referred to the method provided in the embodiments of the present application.

[0064] The non-volatile computer readable storage medium can include a program area and a data area, where the program area can store an operating system and application programs required by at least one function; and the data area can store data created according to the use of the device and the like. In addition, the non-volatile computer readable storage medium can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer readable storage medium can optionally include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0065] The electronic device according to an embodiment of the disclosure can include at least one processor and a memory connected with the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method for multi-modal classification of audio semantic understanding and recognition according to any one of the embodiments of the disclosure.

[0066] The electronic device according to an embodiment of the disclosure can exist in various forms, including but not limited to: (1) Mobile communication device: This kind of device is characterized by having mobile communication function, and providing voice and data communication as the main target. This kind of terminal includes: smart phone, multimedia phone, functional phone, and low-end phone, etc.

[0067] (2) Ultra-mobile personal computer device: This kind of device belongs to the category of personal computer, has the functions of calculation and processing, and generally also has the characteristics of mobile Internet. This kind of terminal includes: PDA, MID and UMPC device, etc., such as tablet computer.

[0068] (3) Portable entertainment device: This kind of device can display and play multimedia content. This kind of device includes: audio and video player, handheld game machine, electronic book, and smart toy and portable car navigation device.

[0069] (4) Other electronic devices with data processing function.

[0070] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises", "comprising", "includes", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0071] The device embodiments described above are only schematic, and the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or can be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0072] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0073] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal classification method for audio semantic understanding and rejection, comprising: Receive the collected raw audio signal, wherein the raw audio signal includes: user dialogue, non-human voices, and environmental noise; The contextual history memory of the original audio signal in voice interaction is obtained, and multi-dimensional features related to semantic understanding and rejection are captured from the original audio signal and the contextual history memory. The contextual history memory includes: the input audio of the voice interaction history dialogue of the original audio signal, and the response result of the input audio. The multi-dimensional features are input into a pre-trained multimodal classification model. The multimodal classification model uses the multi-dimensional features to directly determine the speech recognition content, semantic classification, and rejection result of the original audio signal. The multimodal classification model is an end-to-end architecture. Based on the speech recognition content, semantic classification, and rejection results, the response result for feeding back the original audio signal is determined.

2. The method according to claim 1, wherein, The end-to-end architecture multimodal classification model includes: The audio input module is used to acquire raw audio signals, including user conversations, non-human voices, and ambient noise. A multimodal feature extraction module is used to capture multi-dimensional features of the original audio signal and contextual historical memory. The multi-dimensional features include: speech content, prosodic features reflecting the speaker's expression and emotions, and environmental acoustic features used to understand the audio content and determine rejection. The multimodal large model processing module includes a semantic classification and rejection judgment module to establish the feature structure relationship between multimodal features and speech recognition content, semantic classification and rejection results, and to receive the multi-dimensional features to directly determine the speech recognition content, semantic classification and rejection results of the original audio signal; The result output module is used to determine and output the response results of the speech recognition content, semantic classification, and rejection result.

3. The method according to claim 2, wherein, The training of the end-to-end architecture multimodal classification model includes: The audio input module is used to acquire training audio, including user dialogue, non-human voices and environmental noise, as well as contextual history memory and benchmark multimodal classification labels associated with the training audio. The multimodal features of the training audio are determined by the multimodal feature extraction module. Based on the multimodal large model processing module, a predicted multimodal classification label is determined from the multimodal features, including speech recognition content, semantic classification, and rejection result; Using the benchmark multimodal classification label as a guide, the error between the predicted multimodal classification label and the benchmark multimodal classification label is determined. Based on the error, the end-to-end architecture multimodal classification model is trained until the preset target is reached.

4. The method according to claim 3, wherein, The training of the multimodal classification model of the end-to-end architecture based on the error includes: The audio input module, multimodal feature extraction module, and result output module are frozen, and training is performed only on the parameters within the multimodal large model processing module.

5. The method according to claim 1, wherein, The method further includes: The determined speech recognition content, semantic classification, and rejection results of the original audio signal, as well as the response results of the original audio signal, are used to update the contextual history memory.

6. The method according to claim 1, wherein, If the multimodal classification of the original audio signal is directly determined to be a rejection result, then no response will be given to the original audio signal.

7. A multimodal classification system for audio semantic understanding and rejection, comprising: The acquisition module is used to receive the acquired raw audio signals, wherein the raw audio signals include: user conversations, non-human voices, and environmental noise; The feature extraction module is used to obtain the contextual history memory of the original audio signal in voice interaction, and to capture multi-dimensional features related to semantic understanding and rejection from the original audio signal and the contextual history memory. The contextual history memory includes: the input audio of the voice interaction history dialogue of the original audio signal, and the response result of the input audio. A multimodal classification module is used to input the multi-dimensional features into a pre-trained multimodal classification model. The multimodal classification model uses the multi-dimensional features to directly determine the speech recognition content, semantic classification, and rejection result of the original audio signal. The multimodal classification model is an end-to-end architecture. The result determination module is used to determine the response result of the original audio signal based on the speech recognition content, semantic classification, and rejection result.

8. A storage medium storing a computer program product thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-6.

9. A computer program product having instructions embedded on a storage medium, the instructions implementing the steps of the method according to any one of claims 1-6.

10. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-6.

Images