A voice rejection method, an in-vehicle voice interaction terminal, and a vehicle

Abstract

This application discloses a voice rejection method, an in-vehicle voice interaction system, and a vehicle. The voice rejection method includes: acquiring multimodal perception information; determining the current dialogue scenario in the vehicle based on the multimodal perception information; determining a target rejection threshold based on the current dialogue scenario in the vehicle, and adjusting the rejection threshold of the rejection model to the target rejection threshold; performing rejection judgment based on the rejection model after the rejection threshold adjustment to obtain a recognition result. By comprehensively determining the dialogue scenario in the vehicle through multimodal perception information, the rejection threshold of the rejection model can be adjusted according to the dialogue scenario, enabling the rejection model to dynamically adapt to the dialogue scenario to perform rejection judgment. This effectively improves the recognition accuracy of the rejection model and reduces the occurrence of false wake-up and recognition errors in the voice interaction system in complex dialogue environments.

Description

Technical Field

[0001] This application belongs to the field of human-computer interaction technology, and in particular relates to a voice rejection method, an in-vehicle voice interaction terminal, and a vehicle. Background Technology

[0002] Voice interaction systems are a crucial entry point for automotive intelligent driving and in-vehicle interaction systems to interact with users inside the vehicle. In voice interaction systems, rejection models can help identify and reject inputs that do not belong to any known commands or words, thereby improving the robustness and accuracy of the voice interaction system. However, in-vehicle conversational environments can be quite complex, such as multiple people speaking simultaneously or background noise. These complex environments pose a significant challenge to rejection judgment, making voice interaction systems prone to false wake-ups and recognition errors. Summary of the Invention

[0003] This application provides a voice rejection method, an in-vehicle voice interaction terminal, and a vehicle.

[0004] Firstly, embodiments of this application provide a

[0005] Acquire multimodal sensing information;

[0006] The current dialogue scenario inside the vehicle is determined based on the multimodal perception information;

[0007] The target rejection threshold is determined based on the current dialogue scenario inside the vehicle, and the rejection threshold of the rejection model is adjusted to the target rejection threshold.

[0008] The rejection judgment is performed based on the rejection model after the rejection threshold is adjusted, and the recognition result is obtained.

[0009] In one possible implementation of the first aspect above, determining the current dialogue scenario within the vehicle based on the multimodal perception information includes:

[0010] Scene features are determined based on the multimodal perception information;

[0011] The current dialogue scenario is determined based on the scenario characteristics.

[0012] In one possible implementation of the first aspect above, before performing a rejection judgment based on the rejection model adjusted according to the rejection threshold and obtaining the recognition result, the method further includes:

[0013] Identify the user's dialogue behavior characteristics, and determine the user's interaction tendency based on the user's dialogue behavior characteristics;

[0014] The rejection threshold of the rejection model is adjusted based on the user's interaction tendencies.

[0015] In one possible implementation of the first aspect described above, the multimodal sensing information includes audio sensing information, visual sensing information, and pressure sensing information.

[0016] In one possible implementation of the first aspect described above, the multimodal perception information further includes functional scene perception information.

[0017] In one possible implementation of the first aspect above, the rejection model is a multimodal rejection model.

[0018] In one possible implementation of the first aspect above, the rejection model is connected to a large model of artificial intelligence, and the voice rejection method includes:

[0019] The rejection model performs open-domain dialogue recognition based on the corpus provided by the large model of the artificial intelligence.

[0020] Secondly, embodiments of this application provide an in-vehicle voice interaction system, including:

[0021] The dialogue scene recognition module is used to identify dialogue scenes based on multimodal perception information and determine the current dialogue scene inside the vehicle.

[0022] The rejection module is configured with a rejection model. The rejection module is used to determine the target rejection threshold based on the current dialogue scenario in the vehicle, set the rejection threshold of the rejection model as the target rejection threshold, and perform rejection judgment based on the rejection model adjusted by the rejection parameters to obtain the recognition result.

[0023] Thirdly, embodiments of this application provide an in-vehicle voice interaction terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the voice rejection method provided in any of the first aspects above.

[0024] Fourthly, embodiments of this application provide a vehicle, including a multimodal perception information acquisition device and an in-vehicle voice interaction terminal as described in the third aspect.

[0025] The aforementioned multimodal sensing information acquisition device is used to acquire multimodal sensing information.

[0026] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the voice rejection method provided in any of the first aspects above.

[0027] Sixthly, embodiments of this application provide a computer program product, including a computer program that, when run on a computer, causes the computer to perform the voice rejection method provided in any of the first aspects above.

[0028] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0029] The beneficial effects of the embodiments in this application compared with the prior art are:

[0030] In this embodiment, the dialogue scenario inside the vehicle is comprehensively determined by multimodal perception information. The rejection threshold of the rejection model can be adjusted according to the dialogue scenario, so that the rejection model can dynamically adapt to the dialogue scenario to perform rejection judgment. This can effectively improve the recognition accuracy of the rejection model and reduce the occurrence of false wake-up and recognition errors in the voice interaction system in complex dialogue environments. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the structure of a vehicle provided in one embodiment of this application;

[0032] Figure 2 This is a schematic diagram of the structure of an in-vehicle voice interaction system provided in one embodiment of this application;

[0033] Figure 3 This is a schematic diagram of the structure of another in-vehicle voice interaction system provided in one embodiment of this application;

[0034] Figure 4 This is an example of the application of a vehicle voice interaction system provided in this application, which combines a large artificial intelligence model for semantic recognition.

[0035] Figure 5 This is a schematic diagram of the structure of another in-vehicle voice interaction system provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of the structure of another in-vehicle voice interaction system provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram illustrating the implementation process of a voice rejection method according to an embodiment of this application;

[0038] Figure 8 This is a schematic diagram illustrating the implementation process of another voice rejection method provided in an embodiment of this application;

[0039] Figure 9 This is a schematic diagram illustrating an application scenario of a voice rejection method provided in an embodiment of this application;

[0040] Figure 10 This is an application example of a voice rejection judgment method provided in one embodiment of this application;

[0041] Figure 11 This is a schematic diagram of the structure of an in-vehicle voice interaction terminal provided in one embodiment of this application. Detailed Implementation

[0042] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0043] Voice interaction systems are a crucial entry point for automotive intelligent driving and in-vehicle interaction systems to interact with users inside the vehicle. In voice interaction systems, rejection models can help identify and reject inputs that do not belong to any known commands or words, thereby improving the robustness and accuracy of the voice interaction system. However, in-vehicle conversational environments can be quite complex, such as multiple people speaking simultaneously or background noise. These complex environments pose a significant challenge to rejection judgment, making voice interaction systems prone to false wake-ups and recognition errors.

[0044] Based on this, this application provides a voice rejection method, an in-vehicle voice interaction system, and a vehicle. By comprehensively determining the dialogue scenario in the vehicle through multimodal perception information, the rejection threshold of the rejection model can be adjusted according to the dialogue scenario, enabling the rejection model to dynamically adapt to the dialogue scenario to perform rejection judgment. This can effectively improve the recognition accuracy of the rejection model and reduce the occurrence of false wake-up and recognition errors in the voice interaction system in complex dialogue environments.

[0045] See Figure 1 , Figure 1 A schematic diagram of the structure of a vehicle according to an embodiment of this application is shown, such as... Figure 1 As shown, the vehicle may include a multimodal perception information acquisition device 1 and an in-vehicle voice interaction system 2.

[0046] The multimodal perception information acquisition device 1 may include various in-vehicle information acquisition devices, such as seat pressure sensors, visual perception systems, occupancy monitoring systems (OMS), audio acquisition devices, etc.

[0047] The aforementioned multimodal perception information acquisition device 1 can be used to acquire multimodal perception information, which can be used to identify dialogue scenarios inside the vehicle.

[0048] In practical applications, the aforementioned multimodal sensing information may include, but is not limited to, pressure sensing information, visual sensing information, and audio sensing information.

[0049] In practical applications, the pressure sensing information mentioned above can be collected by the seat pressure sensor, the visual sensing information can be collected by the visual sensing system and the OMS system, and the audio sensing information can be collected by the audio acquisition device.

[0050] In specific applications, the aforementioned audio acquisition device can be a microphone array. The microphone array can have the function of sound source localization, thereby determining audio perception information. This audio perception information can include, but is not limited to, input speech, sound source location, and register occupancy information.

[0051] It is understood that the multimodal perception information in the embodiments of this application may include only audio perception information and visual perception information, or only audio perception information and pressure perception information. Of course, in addition to the above-mentioned audio perception information, pressure perception information and visual perception information, other types of perception information may also be included, such as functional scene perception information.

[0052] See Figure 2 , Figure 2 This application provides a schematic diagram of the structure of an in-vehicle voice interaction system according to an embodiment of the present application. Figure 2 As shown, the in-vehicle voice interaction system 2 may include a dialogue scene recognition module 21 and a rejection module 22. The in-vehicle voice interaction system 2 can communicate with each information acquisition device in the multimodal perception information acquisition device 1, thereby acquiring the multimodal perception information collected by each information acquisition device.

[0053] The dialogue scene recognition module 21 can be used to recognize the dialogue scene based on multimodal perception information and determine the current dialogue scene inside the vehicle. The details of how to recognize the dialogue scene based on multimodal perception information will be discussed in the method embodiments and will not be repeated here.

[0054] The aforementioned rejection module 22 is equipped with a rejection model 221. Based on the rejection model 221, it identifies whether a passenger in the cabin is conversing with the in-vehicle voice interaction system, and responds when it detects that a passenger is conversing with the in-vehicle voice interaction system. That is, the aforementioned rejection model 221 can perform a rejection operation to determine the rejection result.

[0055] In this embodiment of the application, after the dialogue scene recognition module 21 recognizes the current dialogue scene in the vehicle, the rejection module 22 can determine the rejection threshold corresponding to the current dialogue scene in the vehicle, and then adjust the rejection threshold of the rejection model 221 to the rejection threshold (i.e., the target rejection threshold) corresponding to the dialogue scene. Then, the rejection judgment is performed based on the rejection model adjusted by the rejection threshold to obtain the recognition result.

[0056] It should be noted that the rejection model 221 configured in the vehicle voice interaction system 2 can be a commonly used rejection model on the market. This rejection model 221 can perform rejection judgment, perform semantic analysis and intent extraction on the voice input, and then determine whether the input voice is a command voice or a non-command voice.

[0057] Here, command speech refers to the original audio corresponding to the executable voice command received by the system during voice operation. In other words, command speech is the voice that the system receives and must execute or respond to.

[0058] It should be noted that voice commands can be command statements, such as "Open the window," in which case the system can control the car window to open; or they can be expressions that require the system to interpret and respond to, such as "What will the weather be like tomorrow?" in which case the system needs to query relevant weather information and provide feedback, i.e., it can reply with tomorrow's weather conditions based on the queried weather information.

[0059] Non-command voice refers to the raw audio corresponding to voice received by the system but for which no action or response is required, such as the voice of a passenger chatting with the driver.

[0060] In practical applications, the rejection model 221 performs audio conversion, semantic feature extraction, intent classification, and other operations on the input speech, and then outputs the rejection confidence score of the input speech. This rejection confidence score can be a value between 0 and 1. The rejection confidence score is then compared with the rejection threshold. If the rejection confidence score is greater than or equal to the rejection threshold, the recognition result of the input speech as instruction speech is output; if the rejection confidence score is less than the rejection threshold, the recognition result of the input speech as non-instruction speech is output.

[0061] In this embodiment, the aforementioned rejection threshold can be adjusted according to the dialogue scenario. By adjusting the rejection threshold, the rejection model can be made adaptable to complex call scenarios, which can effectively improve the recognition accuracy of the rejection system and reduce the false recognition rate.

[0062] In some embodiments, the rejection model in the rejection module 22 of the above-mentioned vehicle voice interaction system 2 is a multimodal rejection model.

[0063] Among them, the multimodal rejection model can be a rejection model built on a dual-tower architecture that can independently model speech and semantics.

[0064] In practical applications, the two-tower model is a deep learning model structure. Its core idea is to build independent feature representation networks for two entities and perform similarity calculation and matching in a shared semantic space.

[0065] In this embodiment, the two entities selected are the semantic information (text) of the input speech and the acoustic information (audio) of the input speech. The multimodal rejection model can determine the rejection confidence of the input speech and the text corresponding to the input speech identified by the rejection module 22. The rejection confidence is used to characterize the matching accuracy between the identified text and the identified input speech.

[0066] In practical applications, the TinyBERT-6 model can be used to model the semantic information of the input speech, and the Wavevec2-12 model can be used to model the acoustic information of the input speech. Contrastive learning can be introduced in the pre-training and fine-tuning stages, so that the multimodal rejection model can integrate semantic and acoustic information, and the output result will not be affected by a single modality, thereby improving the overall robustness and accuracy of the rejection model.

[0067] In other embodiments, training the above-described multimodal rejection model may further include training each audio... The audio vectors and text vectors of text pairs are compared and learned, and the similarity is calculated by cross-referencing. Audio vectors with a similarity greater than a predetermined similarity threshold are retained. Text pairs are processed, while audio pairs with a similarity score below a predetermined similarity threshold are removed. Text pairs. Specifically, for multimodal rejection models using a dual-encoder structure (dual towers) that jointly models text and speech, there is a certain probability of misidentification in speech recognition (e.g., ASR), such as audio... The text in the text pair is not aligned with the corresponding audio. Multimodal rejection models tend to miss such misaligned audio. The text pair produces incorrect predictions.

[0068] Therefore, through multimodal contrastive learning, the representations of audio and text are mapped to the same semantic space; for example, multiple audio... Text pairs are input into the rejection model in batches, along with audio... The audio and text vectors of the text pair can be processed, for example, by using wav2vec2 to convert each audio data into a set of audio vectors [A1,...,An], and by using GPT2 to convert each text into a set of sentence-level text vectors [T1,...,Tn]. The audio vectors [A1,...,An] and the sentence-level text vectors [T1,...,Tn] are then concatenated to obtain a merged vector [M1,...,Mn]. Then, the audio of the human-computer dialogue is used... Text pairs are used as positive examples (positive training samples), and audio of non-human-computer dialogue is used. Text pairs are used as negative examples (negative training samples). Cross-entropy and loss are calculated for audio vectors, text vectors, and merged vectors respectively, yielding results from multiple supervised learning sessions. Furthermore, audio and text pairs within the same batch are compared and their similarity is calculated cross-referenced. Assuming a batch has n pairs (audio [A1, ..., An], text [T1, ..., Tn]), there are n*n possible combinations (starting from A1). An is a vertical column and T1 is the starting point. Tn is a matrix with rows of n. Contrastive learning can take n combinations from the original dataset (i.e., [A1T1, A2T2, A3T3, ..., AnTn], which is the diagonal of the matrix) as positive samples, and the other n*n combinations as positive samples. n combinations are used as negative samples, and the loss is calculated using, for example, InfoNCE, to obtain the result of contrastive learning. Finally, the four losses (loss for audio vectors, loss for text vectors, loss for merged vectors, and loss for cross-similarity calculation) are weighted to obtain the weighted loss calculation result.

[0069] When using this multimodal rejection model for rejection judgment, the confidence score of the merged vector M after softmax transformation is used as the recognition result of the rejection model. If the current dialogue is predicted to be not a human-computer dialogue (i.e. the recognition result is that the input speech is non-command speech), it needs to be rejected by the vehicle speech recognition system.

[0070] As can be seen from the above, the in-vehicle voice interaction system provided in this application embodiment can adopt a dual-tower architecture to model the semantic information and acoustic information of the input speech respectively, and can introduce contrastive learning in the pre-training stage and the fine-tune stage, so that the multimodal rejection model can integrate semantic and acoustic information, and the output result will not be affected by a single modality, which can effectively improve the overall robustness and accuracy of the rejection model.

[0071] It should be noted that in scenarios or embodiments involving the use of voice interaction systems in vehicles, the training process of the rejection model and / or the use of the rejection model (i.e., obtaining the rejection confidence level) can be carried out in the cloud (e.g., cloud server, cloud platform) to minimize the consumption of vehicle computing resources. Specifically, the vehicle can transmit multimodal perception information, audio data, and recognized text communication to the cloud.

[0072] In other embodiments, the cloud can perform the process of recognizing the dialogue scene and converting audio data into text, while the vehicle only needs to transmit the multimodal perception information and audio data (e.g., audio recorded by the vehicle's microphone) to the cloud. The cloud then sends the recognition result to the vehicle. Of course, it is understood that when the vehicle has sufficient computing resources, the aforementioned dialogue scene recognition and rejection judgment process can also be performed independently by the vehicle. That is, a trained rejection model can be deployed on the vehicle, and then rejection judgment can be performed using the deployed rejection model to obtain the rejection result.

[0073] Please see Figure 3 , Figure 3 This application provides a schematic diagram of the structure of an in-vehicle voice interaction system according to another embodiment. Figure 3 As shown, the in-vehicle voice interaction system 2 provided in this application embodiment may include a dialogue scene recognition module 21, a rejection recognition module 22, and a large artificial intelligence model 23.

[0074] For information on scene recognition module 21 and rejection module 22, please refer to [link / reference]. Figure 1 and Figure 2 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0075] In this embodiment of the application, the large artificial intelligence model 23 can provide a corpus of open-domain dialogue scenarios, providing semantic understanding capabilities of open-domain dialogue for the rejection model 221. This enables the rejection model to perform open-domain dialogue scenario recognition, "non-human speech" filtering, and device orientation judgment based on the semantic information of the input speech, allowing the in-vehicle voice interaction system to handle more complex natural language interaction scenarios.

[0076] Corpus refers to the collection of text, speech, and other data used for artificial intelligence training. Corpus can include a rich variety of data, such as monolingual, bilingual, and multilingual data.

[0077] Large models in artificial intelligence can refer to “large parameter” models trained using large-scale data and powerful computing capabilities. These models typically have high versatility and generalization ability and can be applied to fields such as natural language processing, image recognition, and speech recognition.

[0078] Open-domain dialogue scenarios specifically refer to free dialogue between users and systems without limiting the topic or clear objectives. This form of dialogue is closer to natural human communication, where users can freely express various thoughts and needs, and the system needs to understand and respond based on the user's input.

[0079] "Non-human speech" filtering refers to filtering input speech that lacks specific semantic meaning, as recognized by the in-vehicle voice interaction system. For example, if the in-vehicle voice interaction system receives an input speech that lacks specific semantic meaning, such as "I only know how to wake up this function, is it only for men?", it can recognize this input speech as "non-human speech" and filter it out.

[0080] Device directionality determination can refer to determining whether the input voice is directed at the in-vehicle voice interaction system.

[0081] For example, please refer to Figure 4 , Figure 4 This paper illustrates an application example of the in-vehicle voice interaction system provided in this application, which combines a large-scale artificial intelligence model for semantic recognition. Among other things, Figure 4 (a) in this application is an example of how the in-vehicle interaction system, combined with a large model of artificial intelligence, performs "non-human speech" judgment in the embodiments of this application. Figure 4 As shown in (a), when the content of the input speech is "then because the car didn't have this", the large model of artificial intelligence can determine that the input speech does not have specific semantics. Therefore, the rejection model can identify the input speech as "non-human speech" and thus determine that it is not a command speech, and does not respond. When the content of the input speech is "Which should I choose, Wuling Hongguang or Lamborghini?", the large model of artificial intelligence can determine that the input speech has specific semantics. Therefore, the rejection model can identify that the input speech is not "non-human speech" and thus determine the confidence level of the input speech as a command speech, so as to make a rejection judgment.

[0082] Figure 4 (b) in this embodiment is an application example of the in-vehicle interaction system combined with a large artificial intelligence model to determine device orientation, as provided in this application. Figure 4As shown in (b), when the input voice is "Why are you looking at me like that?", the in-vehicle voice interaction assistant, after semantic judgment based on the large-scale artificial intelligence model, can determine that the input voice is directed at the in-vehicle voice interaction system. Therefore, it can respond, for example, by replying, "Oh, you noticed! I've been silently watching you. I feel you are a very gentle and kind person with a unique charm. Is it because you think I'm interesting?" After receiving the user's input voice as "You are interesting", the in-vehicle voice interaction system, based on semantics, can determine that the input voice is directed at the in-vehicle voice interaction system. Therefore, it can identify the input voice "You are interesting" as the command voice and respond to the command voice.

[0083] Understandably, the large AI model 23 can also provide the rejection model 22 with a judgment on the relevance between the preceding dialogue information and the current dialogue, enabling the rejection model 22 to make rejection judgments by combining contextual semantics, thereby improving the recognition accuracy of the in-vehicle voice interaction system.

[0084] For example, please refer to again Figure 4 , Figure 4 (c) in this application is an example of the in-vehicle interaction system combined with a large model of artificial intelligence to judge the relevance of dialogue.

[0085] like Figure 4 As shown in (c), assuming the historical dialogue was "Turn on the full vehicle ventilation" and the response was "Okay, the full vehicle seat ventilation is on," and the current input voice was "Select this car," then the in-vehicle voice interaction system can determine that the current input voice is unrelated to the previous dialogue. However, assuming the historical dialogue was "Who created you?" and the response was "My parents are the designers who created me; they designed my appearance, taught me to speak, and raised me," and the current input voice was "What is the name of your designer?", then it can be determined that the current input voice is related to the previous dialogue.

[0086] As can be seen from the above, the in-vehicle voice interaction system provided in this application provides the open-domain dialogue semantic understanding capability to the rejection model through the large model of artificial intelligence, enabling the rejection model to perform open-domain dialogue scene recognition, "non-human speech" filtering, and device orientation judgment based on the semantic information of the input speech, thereby enabling the in-vehicle voice interaction system to handle more complex natural language interaction scenarios.

[0087] Please see Figure 5 , Figure 5 This application provides a schematic diagram of the structure of an in-vehicle voice interaction system according to another embodiment. Figure 5As shown, the in-vehicle voice interaction system 2 provided in this application embodiment may include a dialogue scene recognition module 21, a rejection recognition module 22, and a behavior analysis module 24.

[0088] For information on scene recognition module 21 and rejection module 22, please refer to [link / reference]. Figure 1 and Figure 2 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0089] The aforementioned behavior analysis module 24 can be used to identify users' interaction tendencies.

[0090] Accordingly, the aforementioned rejection module 22 can also be used to adjust the rejection threshold of the rejection model 221 according to the user's interaction preferences.

[0091] In practical applications, the behavior analysis module 24 described above can identify users' interaction tendencies in the following ways:

[0092] Method 1: Analyze whether the current round of dialogue is the first round of dialogue.

[0093] In practical applications, if the current conversation is the first round of conversation, the user's tendency to interact will be relatively strong. Therefore, by analyzing whether the conversation is the first round of conversation, the user's interaction tendency can be identified.

[0094] Method 2: Analyze whether the previous round of dialogue was a guiding dialogue.

[0095] In practical applications, if the previous round of dialogue was a guided dialogue, then the user's tendency to interact will be relatively strong.

[0096] Method 3: Analyze whether the previous round of dialogue used a hook question format.

[0097] The "hook question" method refers to posing engaging questions during a conversation to stimulate the other person's desire to share and participate. If the previous round of conversation used the hook question method, the user's tendency to interact will be stronger.

[0098] In practical applications, if a strong tendency to interact with the user is identified, the rejection threshold of the rejection model can be lowered accordingly, that is, the rejection criteria can be relaxed, thereby improving the response efficiency of the rejection model.

[0099] It should be noted that if the behavior analysis module 24 determines, through semantic analysis, wake word analysis, or other methods, that the user's interaction tendency is relatively strong (the interaction tendency meets the adjustment requirements), such as determining that the current dialogue is the first round of dialogue, or that the previous round of dialogue was a guiding dialogue, or that the previous round of dialogue was a hook question, then the rejection module 22 can lower the rejection threshold of the rejection model 221. If the user's interaction tendency does not meet the adjustment requirements (for example, determining that the current dialogue is the first round of dialogue, the previous dialogue was a guiding dialogue, and the previous dialogue was not a hook question), the rejection threshold of the rejection model 221 will be increased, or readjusted to the target rejection threshold.

[0100] In practical applications, the rejection threshold can be lowered by a preset ratio. This preset ratio can be set according to actual application needs. It is understood that the preset ratio can be different in different situations, and this application does not impose any specific restrictions on it.

[0101] As can be seen from the above, the in-vehicle voice interaction system provided in this application embodiment can dynamically adjust the rejection threshold of the rejection model according to the user's interaction tendency, so that the rejection model can dynamically adapt to the user's dialogue behavior, further improve the recognition accuracy of the in-vehicle voice interaction system, reduce false wake-up and false rejection, and enhance the user experience.

[0102] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an in-vehicle voice interaction system provided in another embodiment of this application, as shown below. Figure 6 As shown, the in-vehicle voice interaction system 2 provided in this application embodiment may include a dialogue scene recognition module 21, a rejection recognition module 22, a large artificial intelligence model 23, and a behavior analysis module 24.

[0103] The dialogue scene recognition module 21, rejection module 22, large artificial intelligence model 23, and behavior analysis module 24 in this embodiment can be found in [reference needed]. Figures 1 to 5 The relevant descriptions involved in each embodiment will not be repeated here.

[0104] It is understood that the in-vehicle voice interaction system provided in this application embodiment can integrate the various modules mentioned in the above embodiments, and therefore has the beneficial effects of the in-vehicle voice interaction system described in the above embodiments, which will not be repeated here.

[0105] It should be noted that the information interaction and execution process between the above-mentioned devices are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] The vehicles and in-vehicle voice interaction systems involved in the embodiments of this application have been described above. The voice rejection method provided in the embodiments of this application will be described next:

[0108] Please see Figure 7 , Figure 7 This document illustrates a flowchart of an implementation of a voice rejection method according to an embodiment of this application. The executing entity of this voice rejection method can be an electric vehicle voice interaction terminal equipped with the aforementioned in-vehicle voice interaction system, specifically applicable to the in-vehicle voice interaction system within the in-vehicle voice interaction terminal. The aforementioned electronic device can be, for example... Figure 7 As shown, the voice rejection method provided in this application embodiment may specifically include the following steps:

[0109] In S11, multimodal sensing information is acquired.

[0110] In specific applications, the aforementioned multimodal perception information is collected by the aforementioned multimodal perception information acquisition device 11. The vehicle voice interaction system can obtain the aforementioned multimodal perception information from the multimodal perception information acquisition device 11 in real time or at regular intervals. For example, it can obtain pressure perception information from the seat pressure sensor, visual perception information from the visual perception system and / or OMS system, and audio data such as information from the audio acquisition device.

[0111] In S12, the current dialogue scenario inside the vehicle is determined based on multimodal perception information.

[0112] In practical applications, multimodal perception information can represent scene features such as the number of passengers in the vehicle, seat distribution, posture information, audio zone occupancy information, and the number of people in conversation from different dimensions.

[0113] After acquiring multimodal perception information, the in-vehicle voice interaction system can analyze the aforementioned scene features based on the multimodal perception information, and then make a comprehensive decision based on the scene features to determine the current dialogue scene in the vehicle.

[0114] In practical applications, different perceptual information can reflect different scene characteristics. For example, such as Figure 5 As shown, the aforementioned visual and pressure perception information can reflect scene characteristics such as the number of passengers and seat distribution, while the aforementioned audio perception information can reflect scene characteristics such as wake-up voice range, wake-up method, current voice range, and number of people in the conversation. After obtaining the above scene characteristics, the in-vehicle voice interaction system can determine the current dialogue scene inside the vehicle based on these characteristics.

[0115] Among them, the dialogue scene recognition module can detect the distribution, number, and corresponding actions or postures of passengers in the vehicle in real time based on the visual perception information provided by the OMS system; the dialogue scene recognition module can assist in confirming the number and distribution of passengers in the vehicle based on the pressure perception information provided by the seat pressure sensor; the dialogue scene recognition module can also confirm whether someone is speaking in a certain area of ​​the vehicle, the number of people speaking, etc., based on the audio perception information provided by the microphone array.

[0116] In this application embodiment, the dialogue scenarios inside the vehicle may include, but are not limited to: single-person dialogue scenarios, multi-person dialogue scenarios, intensive dialogue scenarios, multi-party conversation scenarios, etc.

[0117] For example, assuming that the multimodal perception information determines that there is only the driver in the car, the in-vehicle voice interaction system can determine that the current dialogue scenario in the car is a single-person dialogue scenario; and assuming that the multimodal perception information determines that there are 4 people in the car and that there are multiple occupied sound zones, the current dialogue scenario in the car can be determined to be a multi-party conversation scenario.

[0118] By integrating multimodal perception information from different dimensions such as visual perception, pressure perception, and audio perception, the in-vehicle voice recognition system can accurately determine the current dialogue scenario, breaking through the limitation of current in-vehicle voice interaction systems that only perceive information through voice, and providing comprehensive perception capabilities for rejection judgment in complex in-vehicle call scenarios.

[0119] In some embodiments, the rejection model can also combine the voice wake-up word and voice zone occupancy information to determine whether it is necessary to wake up the in-vehicle voice interaction system to respond.

[0120] For example, assuming that the voice wake-up word and voice zone occupancy information determine that the in-vehicle voice interaction system needs to be activated for a response, and multiple command voices are subsequently received, then the current dialogue scenario in the vehicle can be determined to be a dense dialogue scenario. Conversely, assuming that the voice wake-up times and voice zone occupancy information further determine that the in-vehicle voice interaction system does not need to be activated for a response, then the current dialogue scenario in the vehicle can be determined to be a multi-party conversation scenario.

[0121] By accurately determining the wake-up voice zone and dialogue scenario, it can dynamically adapt to different dialogue scenarios, effectively improving the recognition accuracy of the current dialogue scenario in the vehicle.

[0122] In some embodiments, a specific dialogue function mode can be set in the in-vehicle voice interaction system, and the aforementioned multimodal perception information can also include functional scenario perception information.

[0123] In practical applications, the aforementioned dialogue function modes include, but are not limited to, casual conversation mode, vehicle display mode, and role-playing mode.

[0124] Users can set their desired dialogue function mode through the in-vehicle voice interaction system according to their actual needs. After receiving the user's setting operation, the in-vehicle voice interaction system will respond to the setting operation and set the in-vehicle voice interaction system to the corresponding dialogue function mode. The dialogue scene recognition module can combine the perception information of the function scene to determine the current dialogue scene.

[0125] For example, assuming the dialogue function mode of the in-vehicle voice interaction system is the chat mode, the scene recognition module will be more inclined to identify dense dialogue scenes when recognizing the current dialogue scene.

[0126] It should be noted that in-car conversation scenarios can also include various other scenarios, such as one-person conversations with background music, multi-person conversations with background music, multi-person chats with background music, phone calls, multi-person calls, and so on.

[0127] In S13: Determine the target rejection threshold based on the current dialogue scenario inside the vehicle, and adjust the rejection threshold of the rejection model to the target rejection threshold.

[0128] In this embodiment of the application, different rejection thresholds can be set for different dialogue scenarios. After determining the current dialogue scenario in the vehicle, the rejection threshold corresponding to the current dialogue scenario can be determined based on the correspondence between the dialogue scenario and the rejection threshold, and the rejection threshold corresponding to the current dialogue scenario can be determined as the target rejection threshold.

[0129] In practical applications, the rejection threshold for different dialogue scenarios can be set according to actual needs. For example, the rejection threshold for a one-person dialogue scenario can be set to 0.6, and the rejection threshold for a multi-party dialogue scenario can be set to 0.8.

[0130] Understandably, in one-person dialogue scenarios, the likelihood of human-vehicle interaction is higher. Therefore, setting a lower rejection threshold for this dialogue scenario can improve the response speed of the in-vehicle voice interaction system. In multi-party conversation scenarios, since passengers may be chatting, a higher rejection threshold can be set for this dialogue scenario to reduce the false recognition rate.

[0131] It should be noted that the above-mentioned settings for rejection thresholds for different dialogue scenarios are merely examples and not limitations. In actual applications, the rejection thresholds for different dialogue scenarios can be dynamically adjusted based on information such as the accuracy of the recognition results. This application does not impose any specific limitations on this.

[0132] For example, suppose the dialogue scene recognition module in the vehicle voice interaction system recognizes that the current dialogue scene in the vehicle is a one-person dialogue scene, and suppose the rejection threshold corresponding to the one-person dialogue scene is 0.6, then the vehicle voice interaction system can adjust the rejection threshold of the rejection model to 0.6.

[0133] For example, suppose the dialogue scene recognition module in the vehicle voice interaction system recognizes that the current dialogue scene in the vehicle is a multi-party conversation scene, and suppose the rejection threshold corresponding to the multi-party conversation scene is 0.8, then the vehicle voice interaction system can adjust the rejection threshold of the rejection model to 0.8.

[0134] In S14, rejection judgment is made based on the rejection model adjusted by the rejection threshold, and the recognition result is obtained.

[0135] In practical applications, the rejection threshold of the aforementioned rejection model is the standard for the rejection model to distinguish between command speech and non-command speech. When the rejection confidence obtained by the rejection model is greater than or equal to the rejection threshold, the rejection model can output the recognition result that the input speech is command speech. In this way, the in-vehicle voice interaction system can control the corresponding device to perform corresponding actions or answer the questions in the command speech according to the instructions in the command speech. When the rejection confidence is less than the rejection threshold, the rejection model can output the recognition result that the input speech is non-command speech. In this case, the in-vehicle voice interaction system will not respond.

[0136] As can be seen from the above, the voice rejection method provided in this application embodiment can comprehensively determine the dialogue scenario in the vehicle through multimodal perception information, and can adjust the rejection threshold of the rejection model according to the dialogue scenario, so that the rejection model can dynamically adapt to the dialogue scenario to perform rejection judgment, thereby effectively improving the recognition accuracy of the rejection model and reducing the situation of false wake-up and recognition errors in the voice interaction system in complex dialogue environments.

[0137] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating the implementation process of a voice rejection method according to another embodiment of this application, as shown below. Figure 8 As shown, unlike the previous embodiment, the voice rejection method provided in this application embodiment further includes the following steps:

[0138] In S15, the user's dialogue behavior characteristics are identified, and the user's interaction tendency is determined based on the user's dialogue behavior characteristics.

[0139] It should be noted that user dialogue behavior characteristics may include, but are not limited to, whether the dialogue includes a wake word, the current dialogue round, etc.

[0140] In practical applications, in-vehicle voice interaction systems can determine whether the current round of dialogue includes a wake-up word and the current round of dialogue based on historical dialogue content and the content of the current round of dialogue.

[0141] Understandably, if the current round of dialogue includes a wake word, the user's tendency to interact will be stronger; if the current round of dialogue is longer than the preset round, the user's tendency to interact will be weaker.

[0142] In S16, the rejection threshold of the rejection model is adjusted according to the user's interaction preferences.

[0143] In this embodiment of the application, if the user's tendency to interact is relatively strong, the rejection threshold of the rejection model can be lowered; if the user's tendency to interact is relatively weak, the rejection threshold of the rejection model can be raised.

[0144] It should be noted that the rejection threshold adjusted in this embodiment can be the target rejection threshold determined in the previous embodiment. That is, after the in-vehicle voice interaction system determines the target rejection threshold corresponding to the current dialogue scenario, it can first adjust the rejection threshold of the rejection model to the target rejection threshold. During use, it can identify the user's interaction tendency and then adjust the target rejection threshold according to the interaction tendency. For example, if the interaction tendency is strong, the target rejection threshold can be lowered; if the interaction tendency is weak, the target rejection threshold can be raised or readjusted to the target rejection threshold.

[0145] It is understandable that the percentage increase or decrease of the rejection threshold may be the same or different for different dialogue scenarios, and this application does not impose specific restrictions on this.

[0146] In this embodiment, the rejection threshold of the rejection model can be dynamically adjusted according to the user's interaction tendency, so that the rejection model can dynamically adapt to the user's dialogue behavior, further improve the recognition accuracy of the in-vehicle voice interaction system, reduce false wake-up and false rejection, and enhance the user experience.

[0147] To facilitate understanding of this application, specific scenarios are provided below to illustrate the beneficial effects of the voice rejection method provided in the embodiments of this application. Please refer to... Figure 9 , Figure 9 This illustration shows an application scenario diagram of a voice rejection method provided in an embodiment of this application. For example... Figure 9 As shown, the in-vehicle voice interaction system can perform visual detection, audio detection, and seat pressure detection based on multimodal perception information to determine scene features such as the number of passengers, seat distribution, posture information, voice zone occupancy information, and the number of people in the conversation. This allows the system to determine the current dialogue scenario and subsequently the target rejection threshold. Furthermore, the in-vehicle voice interaction system can perform open-domain dialogue scene recognition, "non-human speech" filtering, and device directionality judgment based on a large artificial intelligence model. In addition, the system can dynamically adjust the target rejection threshold based on the user's interaction preferences. The in-vehicle voice interaction system can use a multimodal rejection model to judge the input speech for rejection, calculating text rejection score, acoustic rejection score, multimodal rejection score, and multimodal alignment score. Then, it comprehensively calculates the rejection confidence level and performs rejection judgment based on the adjusted rejection threshold to obtain the rejection result.

[0148] Please participate Figure 10 , Figure 10 Application examples of voice rejection judgment based on the voice rejection method provided in the embodiments of this application are given, such as... Figure 10As shown, when a user inputs "Hi NOMI," the in-vehicle voice interaction system recognizes the wake-up word and responds with "I'm here." Next, when the user inputs "Help me set up an Ultraman ambient light," the system recognizes the input as a command and executes the ambient light setting, replying "Welcome to the world of Ultraman." If the user casually remarks, "Look how nice this ambient light is," the system will recognize this as casual conversation and reject it. However, if the user asks, "How many Ultraman characters do you know?" the system will recognize this as a question and respond with "I know a lot of Ultraman, Ultraman Tiga, Ultraman Jack, Ultraman's Mother…" or "Wow, it knows so many Ultraman!" These responses will be recognized as casual conversation and rejected. "Let's play an Ultraman song," the in-car voice assistant responds to the voice command, plays Ultraman-related music, and replies "Music is here." If it receives the voice input "Exit," it can exit and reply "Okay, see you next time."

[0149] It can be seen that the in-vehicle voice interaction system that applies the voice rejection method provided in the embodiments of this application can accurately identify voice commands and non-voice commands, and can effectively improve the rejection accuracy in complex dialogue environments.

[0150] This application also provides an in-vehicle voice interaction terminal, see [link]. Figure 11 , Figure 11 This application provides a structural block diagram of an in-vehicle voice interaction terminal according to an embodiment of the present application. Figure 11 As shown, this embodiment provides an in-vehicle voice interaction terminal 110, which includes: at least one processor 111, a memory 112, and a computer program 113 stored in the memory 112 and executable on at least one processor 111. When the processor 111 executes the computer program 113, it implements the steps in any of the above method embodiments.

[0151] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in any of the above method embodiments.

[0152] This application provides a computer program product, including a computer program, which, when run on a computer, enables the computer to implement the steps in the various method embodiments described above.

[0153] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a camera device / vehicle voice interaction terminal, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium.

[0154] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0155] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.

[0156] The personal information of users processed by the applicant will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. The applicant will treat the user's personal information and its processing with a high degree of diligence.

[0157] The applicant attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.

Claims

1. A voice rejection method, characterized by, The method comprises: acquiring multi-modal perception information; the multi-modal perception information comprises audio perception information, visual perception information, and pressure perception information; the audio perception information is acquired by an audio collector, the visual perception information is acquired by a visual perception system and an OMS system, and the pressure perception information is acquired by a seat pressure sensor; determining a current conversation scene in the vehicle according to the multi-modal perception information; determining a target rejection threshold according to the current conversation scene in the vehicle, and adjusting a rejection threshold of a rejection model to the target rejection threshold; performing rejection judgment based on the rejection model with the adjusted rejection threshold to obtain a recognition result; the determination of the current conversation scene in the vehicle according to the multi-modal perception information comprises: determining scene features according to the multi-modal perception information; the scene features comprise the number of passengers in the vehicle, seat distribution, posture information, audio zone occupation information, and the number of conversation participants; determining the current conversation scene according to the scene features; the conversation scene comprises a single-person conversation scene, a multi-person conversation scene, a dense conversation scene, and a multi-party conversation scene.

2. The voice recognition method of claim 1, wherein, Before performing rejection judgment based on the rejection model with the adjusted rejection threshold to obtain a recognition result, the method further comprises: recognizing conversation behavior features of a user, and determining an interaction tendency of the user according to the conversation behavior features of the user; adjusting the rejection threshold of the rejection model according to the interaction tendency of the user.

3. The voice recognition method of claim 1, wherein, The multi-modal perception information further comprises functional scene perception information.

4. The voice recognition method according to any one of claims 1 to 3, wherein The rejection model is a multi-modal rejection model.

5. The voice recognition method according to any one of claims 1 to 3, wherein The rejection model is connected to a large model of artificial intelligence, and the speech rejection method comprises: the rejection model performs open-domain conversation recognition based on a corpus provided by the large model of artificial intelligence.

6. A vehicle-mounted voice interaction terminal, characterized by comprising: The vehicle-mounted speech interaction terminal comprises a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the speech rejection method of any one of claims 1 to 5 is implemented.

7. A vehicle characterized by comprising: The vehicle comprises a multi-modal perception information acquisition device and the vehicle-mounted speech interaction terminal of claim 6; the multi-modal perception information acquisition device is configured to acquire multi-modal perception information; the multi-modal perception information comprises audio perception information, visual perception information, and pressure perception information; the audio perception information is acquired by an audio collector, the visual perception information is acquired by a visual perception system and an OMS system, and the pressure perception information is acquired by a seat pressure sensor.

8. A computer program product, characterised in that, The computer program, when executed on a computer, causes the computer to implement the speech rejection method of any one of claims 1 to 5.

Images