Voice rejection method, vehicle-mounted voice interaction terminal and vehicle

Abstract

The invention discloses a voice rejection method, a vehicle-mounted voice interaction system and a vehicle. The voice rejection method comprises the following steps: acquiring multi-mode perception information; determining a current dialogue scene in the vehicle according to the multi-mode perception information; determining a target rejection threshold according to the current dialogue scene in the vehicle, and adjusting the rejection threshold of a rejection model to the target rejection threshold; and performing rejection judgment based on the rejection model after the rejection threshold adjustment to obtain an identification result, comprehensively determining the dialogue scene in the vehicle through the multi-modal sensing information, and adjusting the rejection threshold of the rejection model according to the dialogue scene, so that the rejection model can dynamically adapt to the dialogue scene to execute rejection judgment. Therefore, the recognition accuracy of the rejection model can be effectively improved, and the situations of false wake-up and recognition errors of the voice interaction system in a complex dialogue environment are reduced.

Description

Technical Field

[0001] The present application belongs to the field of human-computer interaction technology, and in particular, relates to a voice rejection method, a vehicle-mounted voice interaction terminal, and a vehicle. Background Art

[0002] The voice interaction system is an important entry point for the intelligent driving and in-vehicle interaction system to interact with the in-vehicle user. In the voice interaction system, the rejection model can help the voice interaction system identify and reject inputs that do not belong to any known commands or vocabulary, thereby improving the robustness and accuracy of the voice interaction system. However, there may be a more complex conversation environment in the car, such as multiple people talking at the same time, background noise, etc. The complex conversation environment brings great challenges to the rejection judgment. In a complex conversation environment, the voice interaction system is prone to problems such as false wake-up and recognition errors. Summary of the invention

[0003] Embodiments of the present application provide a voice rejection method, an in-vehicle voice interaction terminal, and a vehicle.

[0004] In a first aspect, the present application provides a

[0005] Acquire multimodal perception information;

[0006] Determining a current conversation scene in the vehicle based on the multimodal perception information;

[0007] Determine a target rejection threshold according to the current conversation scene in the vehicle, and adjust the rejection threshold of the rejection model to the target rejection threshold;

[0008] A rejection judgment is performed based on the rejection model after the rejection threshold is adjusted to obtain a recognition result.

[0009] In a possible implementation manner of the first aspect, determining a current conversation scene in the vehicle according to the multimodal perception information includes:

[0010] Determining scene features based on the multimodal perception information;

[0011] The current dialogue scene is determined according to the scene features.

[0012] In a possible implementation manner of the first aspect, before performing rejection judgment based on the rejection model after the rejection threshold is adjusted to obtain a recognition result, the method further includes:

[0013] Identifying the user's conversation behavior characteristics, and determining the user's interaction tendency based on the user's conversation behavior characteristics;

[0014] The rejection threshold of the rejection model is adjusted according to the interaction tendency of the user.

[0015] In a possible implementation manner of the first aspect above, the multimodal perception information includes audio perception information, visual perception information, and pressure perception information.

[0016] In a possible implementation manner of the first aspect above, the multimodal perception information also includes functional scenario perception information.

[0017] In a possible implementation manner of the first aspect above, the rejection model is a multimodal rejection model.

[0018] In a possible implementation of the first aspect, the rejection model is connected to a large model of artificial intelligence, and the speech rejection method includes:

[0019] The rejection model performs open domain dialogue recognition based on a corpus provided by the large model of the artificial intelligence.

[0020] In a second aspect, an embodiment of the present application provides a vehicle-mounted voice interaction system, including:

[0021] A dialogue scene recognition module is used to recognize dialogue scenes based on multimodal perception information and determine the current dialogue scene in the car;

[0022] A rejection module is configured with a rejection model, and the rejection module is used to determine a target rejection threshold according to the current dialogue scene in the car, set the rejection threshold of the rejection model as the target rejection threshold, and perform rejection judgment based on the rejection model after the rejection parameters are adjusted to obtain a recognition result.

[0023] In a third aspect, an embodiment of the present application provides an in-vehicle voice interaction terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a voice rejection method as provided in any one of the first aspects above is implemented.

[0024] In a fourth aspect, an embodiment of the present application provides a vehicle, comprising a multimodal perception information collection device and the in-vehicle voice interaction terminal as described in the third aspect.

[0025] The above-mentioned multimodal perception information collection device is used to collect multimodal perception information.

[0026] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the voice rejection method provided in any one of the first aspects above is implemented.

[0027] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is run on a computer, the computer executes the voice rejection method provided in any one of the first aspects above.

[0028] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0029] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0030] In an embodiment of the present application, the conversation scene in the car is comprehensively determined through multimodal perception information, and the rejection threshold of the rejection model can be adjusted according to the conversation scene, so that the rejection model can dynamically adapt to the conversation scene to perform rejection judgment, thereby effectively improving the recognition accuracy of the rejection model and reducing the occurrence of false wake-up and recognition errors in the voice interaction system in complex conversation environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a structural schematic diagram of a vehicle provided by an embodiment of the present application;

[0032] Figure 2 It is a structural diagram of an in-vehicle voice interaction system provided by an embodiment of the present application;

[0033] Figure 3 is a structural diagram of another in-vehicle voice interaction system provided by an embodiment of the present application;

[0034] Figure 4 This is an application example of the in-vehicle voice interaction system provided by an embodiment of the present application combined with a large model of artificial intelligence for semantic recognition;

[0035] Figure 5 It is a structural diagram of another in-vehicle voice interaction system provided by an embodiment of the present application;

[0036] Figure 6 It is a structural diagram of another in-vehicle voice interaction system provided by an embodiment of the present application;

[0037] Figure 7 This is a schematic diagram of an implementation flow of a voice rejection method provided in an embodiment of the present application;

[0038] Figure 8 It is a schematic diagram of the implementation flow of another voice rejection method provided in one embodiment of the present application;

[0039] Fig. 9 This is a schematic diagram of an application scenario of a voice rejection method provided by an embodiment of the present application;

[0040] Fig.10 This is an application example of a speech rejection method for performing rejection judgment provided by an embodiment of the present application;

[0041] Fig.11 It is a structural diagram of a vehicle-mounted voice interaction terminal provided in one embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] The voice interaction system is an important entry point for the intelligent driving and in-vehicle interaction system to interact with the in-vehicle user. In the voice interaction system, the rejection model can help the voice interaction system identify and reject inputs that do not belong to any known commands or vocabulary, thereby improving the robustness and accuracy of the voice interaction system. However, there may be a more complex conversation environment in the car, such as multiple people talking at the same time, background noise, etc. The complex conversation environment brings great challenges to the rejection judgment. In a complex conversation environment, the voice interaction system is prone to problems such as false wake-up and recognition errors.

[0044] Based on this, the present application provides a voice rejection method, an in-vehicle voice interaction system and a vehicle, which comprehensively determine the conversation scene in the car through multimodal perception information, and can adjust the rejection threshold of the rejection model according to the conversation scene, so that the rejection model can dynamically adapt to the conversation scene to perform rejection judgment, thereby effectively improving the recognition accuracy of the rejection model and reducing the occurrence of false wake-up and recognition errors in the voice interaction system in complex conversation environments.

[0045] See also Figure 1 , Figure 1 A schematic diagram of the structure of a vehicle provided by an embodiment of the present application is shown. Figure 1 As shown, the vehicle may include a multimodal perception information collection device 1 and an in-vehicle voice interaction system 2.

[0046] Among them, the multimodal perception information collection device 1 may include a variety of vehicle-mounted information collection devices, such as a seat pressure sensor, a visual perception system, an occupancy monitoring system (OMS), an audio collector, and the like.

[0047] The multimodal perception information collection device 1 can be used to collect multimodal perception information, and the multimodal perception information can be used to identify conversation scenes in the car.

[0048] In practical applications, the above multimodal perception information may include but is not limited to pressure perception information, visual perception information, audio perception information, etc.

[0049] In a specific application, the pressure perception information can be collected through the seat pressure sensor, the visual perception information can be collected through the visual perception system and the OMS system, and the audio perception information can be collected through the audio collector.

[0050] In a specific application, the audio collector may be a microphone array, which may have the function of locating the sound source, and thus may determine the audio perception information, which may include but is not limited to: input voice, sound source location, sound zone occupancy information, and the like.

[0051] It can be understood that the multimodal perception information in the embodiments of the present application may include only audio perception information and visual perception information, or only audio perception information and pressure perception information. Of course, in addition to the above-mentioned audio perception information, pressure perception information and visual perception information, other types of perception information may also be included, such as functional scene perception information, etc.

[0052] See also Figure 2 , Figure 2 FIG. 1 shows a schematic diagram of the structure of a vehicle-mounted voice interaction system provided by an embodiment of the present application. Figure 2 As shown, the vehicle-mounted voice interaction system 2 may include a dialogue scene recognition module 21 and a rejection module 22. The vehicle-mounted voice interaction system 2 may communicate data with each information collection device in the multimodal perception information collection device 1, and thus may acquire the multimodal perception information collected by each information collection device.

[0053] The conversation scene recognition module 21 can be used to recognize the conversation scene according to the multimodal perception information and determine the current conversation scene in the car. The relevant description on how to recognize the conversation scene according to the multimodal perception information will be discussed in detail in the method embodiment and will not be repeated here.

[0054] The rejection module 22 is configured with a rejection model 221, which identifies whether the passenger in the cabin is talking to the vehicle voice interaction system based on the rejection model 221, and responds when it is identified that the passenger is talking to the vehicle voice interaction system. That is, the rejection model 221 can perform a rejection operation to determine a rejection result.

[0055] In an embodiment of the present application, after the conversation scene recognition module 21 recognizes the current conversation scene in the car, the rejection module 22 can determine the rejection threshold corresponding to the conversation scene based on the current conversation scene in the car, and then adjust the rejection threshold of the rejection model 221 to the rejection threshold corresponding to the conversation scene (i.e., the target rejection threshold), and then perform rejection judgment based on the rejection model after the rejection threshold is adjusted to obtain a recognition result.

[0056] It should be noted that the rejection model 221 configured in the in-vehicle voice interaction system 2 can be a commonly used rejection model on the market. The rejection model 221 can perform rejection judgment, perform semantic analysis, intent extraction and other operations on the voice input, and then determine whether the input voice is a command voice or a non-command voice.

[0057] Here, command voice refers to the original audio corresponding to the executable voice command received by the system during voice operation, that is, command voice is the voice received by the system that must be executed or responded to.

[0058] It should be noted that the command voice can be an imperative sentence, for example, if a command voice is "open the window", the system can control the window to open; it can also be an expression of meaning that requires the system to judge and answer, for example, if a command voice is "what will the weather be like tomorrow", the system needs to query relevant weather information and give feedback, that is, it can reply to tomorrow's weather conditions based on the queried weather information.

[0059] Non-command voice refers to the original audio corresponding to the voice received by the system but does not need to be executed or responded to, such as the voice of passengers chatting with drivers.

[0060] In a specific application, the rejection model 221 will perform audio conversion, semantic feature extraction, intent classification, and other operations on the input speech, and then output the rejection confidence of the input speech, which can be a value between 0 and 1. The rejection confidence is then compared with the rejection threshold. When the rejection confidence is greater than or equal to the rejection threshold, the recognition result that the input speech is a command speech is output; when the rejection confidence is less than the rejection threshold, the recognition result that the input speech is a non-command speech is output.

[0061] In the embodiment of the present application, the above-mentioned rejection threshold can be adjusted according to the conversation scenario. The adaptability of the rejection model to complex call scenarios is achieved through the adjustable rejection threshold, which can effectively improve the recognition accuracy of the rejection system and reduce the false recognition rate.

[0062] In some embodiments, the rejection model in the rejection module 22 in the in-vehicle voice interaction system 2 is a multimodal rejection model.

[0063] Among them, the multimodal rejection model can be a rejection model built based on a dual-tower architecture, which can perform independent modeling of speech and semantics.

[0064] In specific applications, the two-tower model is a deep learning model structure. Its core idea is to build independent feature representation networks for two entities and perform similarity calculation and matching in a shared semantic space.

[0065] In the embodiment of the present application, the two entities selected are the semantic information (text) of the input speech and the acoustic information (audio) of the input speech. The multimodal rejection model can determine the rejection confidence of the input speech and the text corresponding to the input speech recognized by the rejection module 22. The rejection confidence is used to characterize the matching accuracy between the recognized text and the recognized input speech.

[0066] In specific applications, the TinyBERT-6 model can be used to model the semantic information of the input speech, and the Wavevec2-12 model can be used to model the acoustic information of the input speech. Contrastive learning can be introduced in the pre-training and fine-tuning stages, so that the multimodal rejection model can integrate semantic and acoustic information while the output result will not be affected by only a single modality, which can improve the overall robustness and accuracy of the rejection model.

[0067] In other embodiments, the training of the multimodal rejection model may further include comparative learning of the audio vector and the text vector of each audio-text pair, and cross-calculating the similarity, retaining the audio-text pairs with a similarity greater than a predetermined similarity threshold, and removing the audio-text pairs with a similarity less than the predetermined similarity threshold. Specifically, for a multimodal rejection model with a dual encoder structure (dual tower) for joint modeling of text and speech, since speech recognition (such as ASR) has a certain probability of misrecognition, for example, the text in the audio-text pair is text that is not aligned with the corresponding audio, the multimodal rejection model is prone to make incorrect predictions for such unaligned audio-text pairs.

[0068] Therefore, through multimodal contrastive learning, the representations of audio and text are mapped to the same semantic space. For example, multiple audio-text pairs are input into the rejection model in batches, and the audio vectors and text vectors of the audio-text pairs are converted into a set of audio vectors [A1,...,An], and each text is converted into a set of sentence-level text vectors [T1,...,Tn] by, for example, wav2vec2 processing, and GPT2 conversion processing, and a set of audio vectors [A1,...,An] and a set of sentence-level text vectors [T1,...,Tn] are concatenated to obtain a set of merged vectors [M1,...,Mn]. Then, the audio-text pairs of human-computer dialogue are used as positive examples (positive training samples), and the audio-text pairs of non-human-computer dialogue are used as negative examples (negative training samples). The cross entropy and loss are calculated for the audio vector, text vector, and merged vector, respectively, to obtain multiple supervised learning results. In addition, the audio and text in the same batch are contrastively learned, and the similarity is cross-calculated respectively. Assuming that a batch has n pairs of (audio [A1,...,An], text [T1,...,Tn]), there can be n*n combinations (a matrix with A1-An as columns and T1-Tn as rows). Contrastive learning can take the n combinations in the original data set (i.e. [A1T1,A2T2,A3T3,...,AnTn], which is the diagonal of the matrix) as positive samples, and the other n*nn combinations as negative samples, and calculate the loss through, for example, InfoNCE to obtain the result of contrastive learning. Finally, the four losses (loss for audio vectors, loss for text vectors, loss for merged vectors, and loss for cross-calculation of similarity) can be weighted to obtain the weighted loss calculation result.

[0069] When using the multimodal rejection model for rejection judgment, the confidence of the merged vector M after softmax conversion is used as the recognition result of the rejection model. If the current dialogue is predicted to be not a human-computer dialogue (that is, the recognition result is that the input voice is non-command voice), it needs to be rejected by the in-vehicle speech recognition system.

[0070] From the above, it can be seen that the in-vehicle voice interaction system provided in the embodiment of the present application can adopt a dual-tower architecture to model the semantic information of the input voice and the acoustic information of the input voice respectively, and can introduce contrastive learning in the pre-training stage and the fine-tune stage, so that the multimodal rejection model can integrate semantic and acoustic information while the output result will not be affected by only a single modality, which can effectively improve the overall robustness and accuracy of the rejection model.

[0071] It should be noted that in the scenario or embodiment of using the voice interaction system for vehicles, the rejection model training process and / or the use of the rejection model (i.e., obtaining the rejection confidence) can be performed in the cloud (e.g., cloud server, cloud platform) to minimize the use of vehicle computing resources. The vehicle side can transmit multimodal perception information, audio data and recognized text communications to the cloud.

[0072] In other embodiments, the cloud can perform the process of identifying the conversation scene and the audio data as text, and the vehicle side only needs to transmit the multimodal perception information and audio data (such as audio recorded by the vehicle microphone) to the cloud. Then the cloud recognition results are transmitted to the vehicle side. Of course, it can be understood that when the computing power resources on the vehicle side are sufficient, the above-mentioned conversation scene recognition and rejection judgment process can also be performed independently by the vehicle side, that is, the trained rejection model is deployed on the vehicle side, and then the rejection judgment is performed through the deployed rejection model to obtain the rejection result.

[0073] See also Figure 3 , Figure 3 A structural diagram of a vehicle-mounted voice interaction system provided in another embodiment of the present application is shown in FIG. Figure 3 As shown, the in-vehicle voice interaction system 2 provided in the embodiment of the present application may include a dialogue scene recognition module 21, a rejection module 22 and a large model 23 of artificial intelligence.

[0074] For details about the scene recognition module 21 and the rejection module 22, see Figure 1 and Figure 2 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0075] In the embodiment of the present application, the large model 23 of artificial intelligence can provide a corpus of open domain dialogue scenarios, and provide the semantic understanding ability of open domain dialogue for the rejection model 221, so that the rejection model can perform open domain dialogue scene recognition, "non-human language" filtering, and device directionality judgment based on the semantic information of the input voice, so that the in-vehicle voice interaction system can handle more complex natural language interaction scenarios.

[0076] Among them, corpus refers to a collection of data such as text and voice used for artificial intelligence training. The corpus can include rich and diverse corpora such as monolingual, bilingual, and multilingual.

[0077] The big models of artificial intelligence23 can refer to "large parameter" models trained using large-scale data and powerful computing power. These models usually have high versatility and generalization capabilities and can be applied to natural language processing, image recognition, speech recognition and other fields.

[0078] Open domain dialogue scenarios specifically refer to free dialogues between users and systems without a defined topic or clear goal. This form of dialogue is closer to natural human communication, where users can freely express various ideas and needs, and the system needs to understand and respond based on the user's input.

[0079] "Non-human speech" filtering may refer to filtering input speech that does not have specific semantics recognized by the in-vehicle voice interaction system. For example, when the in-vehicle voice interaction system receives an input speech that does not have specific semantics, such as "I can only wake up this function with a well-known name. Is it only for men?", the input speech may be recognized as "non-human speech" and filtered.

[0080] Device directionality judgment may refer to judging whether the input voice is directed to the in-vehicle voice interaction system.

[0081] For example, see Figure 4 , Figure 4 The example of using the vehicle voice interaction system provided by the embodiment of the present application in combination with a large model of artificial intelligence for semantic recognition is shown. Figure 4 (a) is an application example of the in-vehicle interactive system provided in the embodiment of the present application combined with a large model of artificial intelligence to perform "non-human speech" judgment, such as Figure 4 As shown in (a), when the content of the input voice is "Then because the car is not for this", the large model of artificial intelligence can determine that the input voice does not have specific semantics. Therefore, the rejection model can recognize that the input voice is "non-human language", and then it can be determined that the non-command voice does not respond; when the content of the input voice is "Which one should I choose between Wuling Hongguang and Lamborghini", the large model of artificial intelligence can determine that the input voice has specific semantics. Therefore, the rejection model can recognize that the input voice is not "non-human language", and then the rejection model can determine the confidence that the input voice is a command voice to make a rejection judgment.

[0082] Figure 4 (b) is an application example of the in-vehicle interactive system provided by the embodiment of the present application combined with a large model of artificial intelligence to determine the directionality of the device. Figure 4As shown in (b) in the figure, when the content of the input voice is "Why are you looking at me like that", the in-vehicle voice interaction assistant can determine that the input voice is issued to the in-vehicle voice interaction system after combining the large model of artificial intelligence with its semantic judgment. Therefore, it can respond, for example, replying "Oh, you can see that I have been silently paying attention to you. I feel that you are a gentle, kind, and uniquely charming person. Is it because you think I am interesting?" After receiving the input voice input by the user as "You are interesting", the in-vehicle voice interaction system can determine that the input voice is issued to the in-vehicle voice interaction system based on the semantics. Therefore, the input voice of "You are interesting" can be determined as a command voice and respond to the command voice.

[0083] It is understandable that the artificial intelligence large model 23 can also provide the rejection model 22 with a relevance judgment between the previous conversation information and the current conversation, so that the rejection model 22 can combine the contextual semantics to make a rejection judgment, thereby improving the recognition accuracy of the in-vehicle voice interaction system.

[0084] For example, please refer again Figure 4 , Figure 4 (c) is an application example of the in-vehicle interaction system provided in an embodiment of the present application combined with a large model of artificial intelligence to judge the relevance of dialogue.

[0085] like Figure 4 As shown in (c) in the figure, assuming that the historical dialogue is "Turn on the ventilation of the whole car", "OK, the ventilation of all seats in the car is turned on", and the current input voice is "Choose this car", the in-vehicle voice interaction system can determine that the current input voice is not related to the previous dialogue. Assuming that the historical dialogue is "Who made you?", "My parents are the designers who created me. They designed my appearance, taught me to speak, and helped me grow up", and the current input voice is "What is the name of your designer", it can be determined that the current input voice is related to the previous dialogue.

[0086] From the above, it can be seen that the in-vehicle voice interaction system provided in the embodiment of the present application provides the rejection model with the semantic understanding ability of open domain dialogue through the large model of artificial intelligence, so that the rejection model can perform open domain dialogue scene recognition, "non-human language" filtering, and device directionality judgment based on the semantic information of the input voice, thereby enabling the in-vehicle voice interaction system to handle more complex natural language interaction scenarios.

[0087] See also Figure 5 , Figure 5 A structural diagram of a vehicle-mounted voice interaction system provided in another embodiment of the present application is shown in FIG. Figure 5As shown, the in-vehicle voice interaction system 2 provided in the embodiment of the present application may include a dialogue scene recognition module 21, a rejection module 22 and a behavior analysis module 24.

[0088] For details about the scene recognition module 21 and the rejection module 22, see Figure 1 and Figure 2 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0089] The above-mentioned behavior analysis module 24 can be used to identify the user's interaction tendency.

[0090] Correspondingly, the rejection module 22 may also be used to adjust the rejection threshold of the rejection model 221 according to the user's interaction tendency.

[0091] In a specific application, the behavior analysis module 24 can identify the user's interaction tendency in the following ways:

[0092] Method 1: Analyze whether the current round of dialogue is the first round of dialogue.

[0093] In actual applications, if the current conversation is the first round of conversation, the user's interaction tendency will be stronger. Therefore, by analyzing whether the conversation is the first round of conversation, the user's interaction tendency can be identified.

[0094] Method 2: Analyze whether the previous round of dialogue was a guided dialogue.

[0095] In actual applications, if the previous round of dialogue is a guided dialogue, the user's interaction tendency will be stronger.

[0096] Method 3: Analyze whether the previous round of dialogue was a hook question.

[0097] Among them, the hook question method refers to stimulating the other party's desire to share and participate by asking attractive questions in the conversation. If the previous round of conversation was a hook question method, the user's interaction tendency will be stronger.

[0098] In specific applications, if it is identified that the interaction tendency with the user is relatively strong, the rejection threshold of the rejection model can be lowered accordingly, that is, the rejection standard can be relaxed, thereby improving the response efficiency of the rejection model.

[0099] It should be noted that if the behavior analysis module 24 determines that the user's interaction tendency is relatively strong (the interaction tendency meets the adjustment requirements) through semantic analysis, wake-up word analysis, etc., for example, if the current dialogue is determined to be the first round of dialogue, or the previous dialogue is determined to be an introductory dialogue, or the previous dialogue is determined to be a hook question, then the rejection module 22 can lower the rejection threshold of the rejection model 221. If the user's interaction tendency does not meet the adjustment requirements (for example, if the current dialogue is determined to be the first round of dialogue, the previous dialogue is the introductory dialogue, and the previous dialogue is not a hook question), the rejection threshold of the rejection model 221 is increased, or readjusted to the target rejection threshold.

[0100] In actual application, the rejection threshold can be lowered by a preset ratio, and the preset ratio can be set according to actual application requirements. It can be understood that the preset ratio in different situations may be different, and this application does not impose specific restrictions on this.

[0101] From the above, it can be seen that the in-vehicle voice interaction system provided in the embodiment of the present application can dynamically adjust the rejection threshold of the rejection model according to the user's interaction tendency, so that the rejection model can dynamically adapt to the user's dialogue behavior, further improve the recognition accuracy of the in-vehicle voice interaction system, reduce false wake-up and false rejection, and enhance the user experience.

[0102] See also Figure 6 , Figure 6 A structural diagram of a vehicle-mounted voice interaction system provided in another embodiment of the present application is shown in FIG. Figure 6 As shown, the in-vehicle voice interaction system 2 provided in the embodiment of the present application may include a dialogue scene recognition module 21, a rejection module 22, an artificial intelligence large model 23 and a behavior analysis module 24.

[0103] The dialogue scene recognition module 21, the rejection module 22, the artificial intelligence large model 23 and the behavior analysis module 24 in the embodiment of the present application can be found in Figures 1 to 5 The relevant descriptions involved in each embodiment will not be repeated in this application.

[0104] It can be understood that the in-vehicle voice interaction system provided in the embodiment of the present application can integrate the various modules mentioned in the above-mentioned embodiments, and therefore has the beneficial effects of the in-vehicle voice interaction system described in the above-mentioned embodiments, which will not be repeated here.

[0105] It should be noted that the information interaction, execution process and other contents between the above-mentioned devices are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0106] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0107] The above describes the vehicle and the vehicle-mounted voice interaction system involved in the embodiment of the present application. Next, the voice rejection method provided in the embodiment of the present application is described:

[0108] See also Figure 7 , Figure 7 The flowchart of a method for rejecting voice recognition provided by an embodiment of the present application is shown. The execution subject of the method for rejecting voice recognition may be an electric vehicle voice interaction terminal equipped with the above-mentioned vehicle voice interaction system, and may be specifically applied to the vehicle voice interaction system in the vehicle voice interaction terminal. The above-mentioned electronic device may be, for example, Figure 7 As shown, the voice rejection method provided in the embodiment of the present application may specifically include the following steps:

[0109] In S11, multimodal perception information is obtained.

[0110] In specific applications, the above-mentioned multimodal perception information is collected by the above-mentioned multimodal perception information collection device 11, and the vehicle-mounted voice interaction system can obtain the above-mentioned multimodal perception information from the multimodal perception information collection device 11 in real time or periodically, for example, pressure perception information can be obtained from the seat pressure sensor, visual perception information can be obtained from the visual perception system and / or OMS system, and audio data such as information can be collected from the audio collector.

[0111] In S12, the current conversation scene in the vehicle is determined based on the multimodal perception information.

[0112] In specific applications, multimodal perception information can represent scene characteristics such as the number of passengers in the car, seat distribution, posture information, sound zone occupancy information, number of people in conversation, etc. from different dimensions.

[0113] After obtaining multimodal perception information, the in-vehicle voice interaction system can analyze the above-mentioned scene characteristics based on the multimodal perception information, and then make comprehensive decisions based on the scene characteristics to determine the current dialogue scene in the car.

[0114] In specific applications, different perception information can reflect different scene characteristics. For example, Figure 5 As shown, the visual perception information and pressure perception information can reflect the scene characteristics of the number of passengers and seat distribution, and the audio perception information can reflect the scene characteristics of the wake-up sound zone, wake-up method, current sound zone, number of people in conversation, etc. After obtaining the above scene characteristics, the in-vehicle voice interaction system can determine the current conversation scene in the car according to the various scene characteristics.

[0115] Among them, the dialogue scene recognition module can detect the distribution of passengers in the car, the number of people and the corresponding actions or postures in real time according to the visual perception information provided by the OMS system; the dialogue scene recognition module can assist in confirming the number and distribution of passengers in the car according to the pressure perception information provided by the seat pressure sensor; the dialogue scene recognition module can also confirm whether there is someone speaking in a certain area of ​​the car, the number of people speaking, etc. according to the audio perception information provided by the microphone array.

[0116] In the embodiment of the present application, the conversation scenes in the car may include but are not limited to: single-person conversation scenes, multi-person conversation scenes, intensive conversation scenes, multi-party conversation scenes, etc.

[0117] For example, assuming that it is determined based on multimodal perception information that there is only a driver in the car, the in-vehicle voice interaction system can determine that the current conversation scene in the car is a single-person conversation scene; assuming that it is determined based on multimodal perception information that there are four people in the car and there are currently multiple occupied sound zones, it can be determined that the current conversation scene in the car is a multi-party conversation scene.

[0118] By integrating multimodal perception information from different dimensions such as visual perception, pressure perception, and audio perception, the in-vehicle voice recognition system can accurately judge the current conversation scenario, breaking through the limitation of the current in-vehicle voice interaction system that only perceives information through voice, and providing comprehensive perception capabilities for rejection judgment in complex call scenarios in the car.

[0119] In some embodiments, the rejection model can also combine the voice wake-up word and the sound zone occupancy information to determine whether it is necessary to wake up the in-vehicle voice interaction system to respond.

[0120] For example, if the combination of the voice wake-up word and the audio zone occupancy information determines that the vehicle voice interaction system needs to be awakened for response, and then multiple command voices are received, the current conversation scene in the car can be determined to be an intensive conversation scene. If the combination of the voice wake-up times and the audio zone occupancy newly determines that the vehicle voice interaction system does not need to be awakened for response, the current conversation scene in the car can be determined to be a multi-party conversation scene.

[0121] Through accurate judgment of the wake-up sound zone and dialogue scene, it can dynamically adapt to different dialogue scenes and effectively improve the recognition accuracy of the current dialogue scene in the car.

[0122] In some embodiments, a specific dialogue function mode can also be set in the in-vehicle voice interaction system, and the above-mentioned multimodal perception information can also include functional scene perception information.

[0123] In specific applications, the above-mentioned dialogue function modes include but are not limited to chat mode, car exhibition mode and role-playing mode.

[0124] Users can set the required dialogue function mode through the in-vehicle voice interaction system according to actual needs. After receiving the user's setting operation, the in-vehicle voice interaction system will respond to the setting operation and set the in-vehicle voice interaction system to the corresponding dialogue function mode. The dialogue scene recognition module can determine the current dialogue scene in combination with the function scene perception information.

[0125] For example, assuming that the conversation function mode of the in-vehicle voice interaction system is the chat mode, the scene recognition module will be more inclined to the intensive conversation scene when identifying the current conversation scene.

[0126] It should be noted that the conversation scenes in the car may also include various other scenes, such as a single-person conversation scene with background music, a multi-person conversation scene with background music, a multi-person chat scene with background music, a call scene, a multi-person call scene, and the like.

[0127] In S13: a target rejection threshold is determined according to the current conversation scene in the vehicle, and the rejection threshold of the rejection model is adjusted to the target rejection threshold.

[0128] In an embodiment of the present application, different rejection thresholds can be set for different conversation scenarios. After determining the current conversation scene in the car, the rejection threshold corresponding to the current conversation scene can be determined based on the correspondence between the conversation scene and the rejection threshold, and the rejection threshold corresponding to the current conversation scene can be determined as the target rejection threshold.

[0129] In specific applications, the rejection thresholds for different conversation scenarios can be set according to actual needs. For example, the rejection threshold for a single-person conversation scenario can be set to 0.6, and the rejection threshold for a multi-party conversation scenario can be set to 0.8.

[0130] It is understandable that in a single-person conversation scenario, the possibility of human-vehicle interaction is higher. Therefore, setting the rejection threshold corresponding to the conversation scenario relatively low can improve the response speed of the in-vehicle voice interaction system. In a multi-party conversation scenario, since passengers may be chatting with each other, the rejection threshold corresponding to the conversation scenario can be set relatively high to reduce the misrecognition rate.

[0131] It should be noted that the above settings of the rejection thresholds corresponding to different dialogue scenarios are only examples and not limitations. In actual application, the values ​​of the rejection thresholds corresponding to different dialogue scenarios can be dynamically adjusted based on information such as the accuracy of the recognition results. This application does not impose any specific restrictions on this.

[0132] For example, assuming that the conversation scene recognition module in the in-vehicle voice interaction system recognizes that the current conversation scene in the car is a single-person conversation scene, and assuming that the rejection threshold corresponding to the single-person conversation scene is 0.6, the in-vehicle voice interaction system can adjust the rejection threshold of the rejection model to 0.6.

[0133] As another example, assuming that the conversation scene recognition module in the in-vehicle voice interaction system recognizes that the current conversation scene in the car is a multi-party conversation scene, and assuming that the rejection threshold corresponding to the multi-party conversation scene is 0.8, the in-vehicle voice interaction system can adjust the rejection threshold of the rejection model to 0.8.

[0134] In S14, a rejection judgment is performed based on the rejection model after the rejection threshold is adjusted to obtain a recognition result.

[0135] In specific applications, the rejection threshold of the above-mentioned rejection model is the standard for the rejection model to identify command voice and non-command voice. When the rejection confidence obtained by the rejection model is greater than or equal to the rejection threshold, the rejection model can output the recognition result that the input voice is a command voice, so that the in-vehicle voice interaction system can control the corresponding device to perform corresponding actions according to the instructions in the command voice, or answer the questions in the command voice; when the rejection confidence is less than the rejection threshold, the rejection model can output the recognition result that the input voice is a non-command voice, and the in-vehicle voice interaction system will not respond.

[0136] From the above, it can be seen that a voice rejection method provided in an embodiment of the present application can comprehensively determine the conversation scene in the car through multimodal perception information, and can adjust the rejection threshold of the rejection model according to the conversation scene, so that the rejection model can dynamically adapt to the conversation scene to perform rejection judgment, thereby effectively improving the recognition accuracy of the rejection model and reducing the occurrence of false wake-up and recognition errors in the voice interaction system in a complex conversation environment.

[0137] See also Figure 8 , Figure 8 A flowchart of a method for implementing a voice rejection method provided in another embodiment of the present application is shown in FIG. Figure 8 As shown, different from the previous embodiment, the voice rejection method provided in the embodiment of the present application further includes the following steps:

[0138] In S15, the user's dialogue behavior characteristics are identified, and the user's interaction tendency is determined according to the user's dialogue behavior characteristics.

[0139] It should be noted that the user's conversation behavior characteristics may include but are not limited to whether the conversation includes a wake-up word, the current conversation turn, etc.

[0140] In specific applications, the in-vehicle voice interaction system can determine whether the current round of conversation includes a wake-up word, the current round of conversation, and other conversation behavior characteristics based on the historical conversation content and the current round of conversation content.

[0141] It is understandable that if the current round of conversation includes a wake-up word, the user's interaction tendency will be stronger, and if the current round of conversation is greater than the preset round, the user's interaction tendency will be weaker.

[0142] In S16, the rejection threshold of the rejection model is adjusted according to the user's interaction tendency.

[0143] In the embodiment of the present application, if the user's interaction tendency is relatively strong, the rejection threshold of the rejection model can be lowered; if the user's interaction tendency is relatively weak, the rejection threshold of the rejection model can be increased.

[0144] It should be noted that the rejection threshold adjusted in the embodiment of the present application can specifically be the target rejection threshold determined in the previous embodiment, that is, after the in-vehicle voice interaction system determines the target rejection threshold corresponding to the current dialogue scenario, it can first adjust the rejection threshold of the rejection model to the target rejection threshold, and during use, identify the user's interaction tendency, and then adjust the target rejection threshold according to the interaction tendency. For example, when the interaction tendency is stronger, lower the target rejection threshold, and when the interaction tendency is weaker, increase the target rejection threshold or readjust it to the target rejection threshold, etc.

[0145] It is understandable that, corresponding to different dialogue scenarios, the ratio of raising or lowering the rejection threshold may be the same or different, and this application does not impose any specific limitation on this.

[0146] In an embodiment of the present application, the rejection threshold of the rejection model can be dynamically adjusted according to the user's interaction tendency, so that the rejection model can dynamically adapt to the user's conversational behavior, further improving the recognition accuracy of the in-vehicle voice interaction system, reducing false wake-ups and false rejections, and enhancing the user experience.

[0147] In order to facilitate understanding of the present application, the following examples are given in conjunction with specific scenarios to illustrate the beneficial effects of the voice rejection method provided in the present application embodiment. Fig. 9 , Fig. 9 FIG. 1 is a schematic diagram showing an application scenario of a voice rejection method provided by an embodiment of the present application. Fig. 9 As shown, the in-vehicle voice interaction system can perform visual detection, audio detection and seat pressure detection based on multimodal perception information to determine scene features such as the number of passengers, seat distribution, posture information, sound zone occupancy information, and the number of people in conversation, and then determine the current conversation scene, and then determine the target rejection threshold. The in-vehicle voice interaction system can perform open domain conversation scene recognition, "non-human speech" filtering, and device directionality judgment based on the large model of artificial intelligence. In addition, the in-vehicle voice interaction system can also dynamically adjust the target rejection threshold according to the user's interaction tendency. The in-vehicle voice interaction system can use the multimodal rejection model to reject the input voice, that is, it can calculate the text rejection score, acoustic rejection score, multimodal rejection score, and multimodal alignment score, and then comprehensively calculate the rejection confidence, and perform rejection judgment based on the adjusted rejection threshold to obtain a rejection result.

[0148] Please participate Fig.10 , Fig.10 An application example of performing rejection judgment based on the speech rejection method provided in the embodiment of the present application is given, such as Fig.10As shown, the user inputs "Hi NOMI", the in-vehicle voice interaction system can determine that the user has input the wake-up word, so it responds to the user and replies "I'm here", and then the user inputs "Help me set up an Ultraman atmosphere light", the in-vehicle voice interaction system can determine that the input voice input by the user is a command voice, so it performs the operation of setting the atmosphere light and replies "Welcome to the world of Ultraman", and the user chats with each other "You see, this atmosphere light is pretty good", the in-vehicle voice interaction system can recognize the input voice as user chat and refuse to recognize it. "How many kinds of Ultraman do you know", the in-vehicle voice interaction system can recognize the input voice as the user's inquiry sentence, that is, it can respond and reply "I know a lot of Ultraman, Ultraman Tiga, Ultraman Jack, Mother of Ultraman..."; "Wow, it actually knows so many Ultraman", which can be recognized as user chat and refused to recognize it. "Let's play an Ultraman song", the in-car voice interaction assistant responds to the voice command, plays Ultraman-related music, and replies "The music is coming". If it receives the input voice of "exit", it can execute the exit operation and reply "OK, see you next time."

[0149] It can be seen from this that the in-vehicle voice interaction system that applies the voice rejection method provided in the embodiment of the present application can accurately identify voice commands and non-voice commands, and can effectively improve the rejection accuracy in complex dialogue environments.

[0150] The present application also provides a vehicle-mounted voice interaction terminal. Fig.11 , Fig.11 A structural block diagram of a vehicle-mounted voice interaction terminal provided by an embodiment of the present application is shown. Fig.11 As shown, this embodiment provides an in-vehicle voice interaction terminal 110, which includes: at least one processor 111, a memory 112, and a computer program 113 stored in the memory 112 and executable on at least one processor 111. When the processor 111 executes the computer program 113, the steps in any of the above-mentioned method embodiments are implemented.

[0151] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments can be implemented.

[0152] An embodiment of the present application provides a computer program product, including a computer program. When the computer program product is run on a computer, the computer can implement the steps of the above-mentioned method embodiments when executing the computer.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the camera device / vehicle voice interaction terminal, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium.

[0154] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0155] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0156] The user personal information processed by the applicant will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service, which may involve the user's account information, device information, driving information, vehicle information or other related information. The applicant will treat the user's personal information and its processing with a high degree of diligence.

[0157] The Applicant attaches great importance to the security of user personal information and has adopted reasonable and feasible security protection measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

Claims

1. A method for rejecting speech recognition, characterized in that: include: Acquire multimodal perception information; Determining a current conversation scene in the vehicle based on the multimodal perception information; Determine a target rejection threshold according to the current conversation scene in the vehicle, and adjust the rejection threshold of the rejection model to the target rejection threshold; A rejection judgment is performed based on the rejection model after the rejection threshold is adjusted to obtain a recognition result.

2. The voice rejection method according to claim 1, characterized in that: The determining the current conversation scene in the vehicle according to the multimodal perception information includes: Determining scene features based on the multimodal perception information; The current dialogue scene is determined according to the scene features.

3. The method for rejecting speech recognition according to claim 1, wherein: Before performing a rejection judgment based on the rejection model after the rejection threshold is adjusted to obtain a recognition result, the method further includes: Identifying the user's conversation behavior characteristics, and determining the user's interaction tendency based on the user's conversation behavior characteristics; The rejection threshold of the rejection model is adjusted according to the interaction tendency of the user.

4. The method for rejecting speech recognition as claimed in claim 2, characterized in that: The multimodal perception information includes audio perception information, visual perception information and pressure perception information.

5. The method for rejecting speech recognition as claimed in claim 4, characterized in that: The multimodal perception information also includes functional scenario perception information.

6. The voice rejection method according to any one of claims 1 to 5, characterized in that: The rejection model is a multimodal rejection model.

7. The voice rejection method according to any one of claims 1 to 5, characterized in that: The rejection model is connected to a large model of artificial intelligence, and the speech rejection method includes: The rejection model performs open domain dialogue recognition based on a corpus provided by the large model of the artificial intelligence.

8. A vehicle-mounted voice interaction terminal, characterized in that: The in-vehicle voice interaction system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the voice rejection method as described in any one of claims 1 to 7 is implemented.

9. A vehicle, characterized in that: The vehicle comprises a multimodal perception information collection device and the vehicle-mounted voice interaction terminal according to claim 8; The multimodal perception information collection device is used to collect multimodal perception information.

10. A computer program product, characterized in that It comprises a computer program, and when the computer program is run on a computer, the computer is enabled to implement the speech rejection method as claimed in any one of claims 1 to 7.

Images