Vehicle cockpit voice intent recognition methods, devices, and vehicle control methods

Abstract

This application discloses a method, device, and vehicle control method for vehicle cockpit voice intent recognition. It involves acquiring the user's current voice input text data, the user's current facial expression and body language data, and vehicle internal and external environmental data; using a multi-classification model to perform user intent recognition on the current voice input text data to obtain a first intent recognition result; and using a multi-modal recognition model to perform user intent recognition based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and vehicle internal and external environmental data to obtain a second intent recognition result; and fusing the first and second intent recognition results to obtain the true user intent recognition result. By fusing the intent recognition results from the multi-classification model and the multi-modal recognition model, the accuracy of voice intent understanding can be improved, thereby enhancing the intelligence of the vehicle cockpit dialogue system and the user experience.

Description

Technical Field

[0001] This application relates to the field of intelligent vehicle technology, and in particular to a method, device and vehicle control method for recognizing voice intent in a vehicle cockpit. Background Technology

[0002] The vehicle cabin is the part of the vehicle where the driver and passengers interact most closely. With the continuous development of intelligent cockpits and autonomous driving technology, vehicles are no longer just tools to assist people in traveling, but the vehicle cabin is gradually becoming a part of people's diversified living space, making human-computer interaction in the vehicle cabin particularly important.

[0003] Currently, the most common way of human-machine interaction in vehicle cockpits is voice interaction, where drivers and passengers send commands to the vehicle cockpit via voice. After processing through voice recognition, semantic understanding, and speech synthesis, the vehicle cockpit engages in dialogue with the user to complete the recognition of the user's intent and executes vehicle cockpit control based on the recognized user intent.

[0004] This method of intent recognition relies on the driver's and passengers' current voice data to identify user intent. However, because the current voice data is too concise and lacks necessary information, it cannot accurately identify user intent, thus affecting the intelligence of the cockpit human-computer interaction and the user experience. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a vehicle cockpit voice intent recognition method, device, vehicle control method model training method, training device, and medical image classification method, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.

[0006] To achieve the above and other related objectives, this application provides a vehicle cockpit voice intent recognition method, comprising:

[0007] Acquire the user's current voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data;

[0008] A multi-classification model is used to perform user intent recognition on the user's current voice input text data to obtain a first intent recognition result;

[0009] Using a multimodal recognition model, user intent is identified based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data, in order to obtain a second intent recognition result.

[0010] The first intent recognition result and the second intent recognition result are fused together to obtain the user's true intent recognition result.

[0011] In an optional embodiment of this application, the user's current voice input text data is saved as the user's historical voice input text data.

[0012] In an optional embodiment of this application, fusing the first intent recognition result with the second intent recognition result to obtain the user's true intent further includes:

[0013] Determine whether additional entity information is needed based on the intent category of the first intent recognition result;

[0014] When it is necessary to supplement entity information, entity information corresponding to the first intent recognition result is extracted from the user's current voice input text data.

[0015] The first intent recognition result, the entity information, and the second intent recognition result are fused together to obtain the user's true intent recognition result.

[0016] In an optional embodiment of this application, obtaining the user's current voice input text data includes:

[0017] Obtain the user's current voice input audio data;

[0018] The speech recognition module performs content recognition on the user's current voice input audio data to obtain the user's current voice input text data.

[0019] In one optional embodiment of this application, the vehicle internal and external environmental data includes at least one of vehicle internal temperature data, vehicle internal air pressure data, vehicle external temperature data, and weather conditions.

[0020] In one optional embodiment of this application, the multi-classification model includes a decision tree model or a deep learning model; the multimodal recognition model includes a CogVLM model or a GPT model.

[0021] In one optional embodiment of this application, the multimodal recognition model is set in the cloud.

[0022] In an optional embodiment of this application, the first intent recognition result includes a first predicted intent and a corresponding predicted probability, and the second intent recognition result includes a second predicted intent and a corresponding predicted probability.

[0023] The first intent recognition result and the second intent recognition result are fused to obtain the user's true intent recognition result, including:

[0024] When the predicted probability of the first predicted intent is greater than or equal to the first preset threshold, the first predicted intent and the corresponding predicted probability are taken as the user's true intent recognition result.

[0025] When the prediction probability of the first predicted intent is less than the first preset threshold, and the prediction probability of the second predicted intent is greater than or equal to the second preset threshold, the second predicted intent and its corresponding prediction probability are taken as the user's true intent recognition result.

[0026] When the prediction probability of the first predicted intent is less than the first preset threshold, the prediction probability of the second predicted intent is less than the second preset threshold, and the first predicted intent and the second predicted intent are the same, probability fusion is performed to enhance the prediction probability of the first predicted intent or the second predicted intent, and the first predicted intent and the enhanced prediction probability or the second predicted intent and the enhanced prediction probability are used as the user's true intent recognition result.

[0027] When the prediction probability of the first predicted intent is less than the first preset threshold, the prediction probability of the second predicted intent is less than the second preset threshold, and the first predicted intent and the second predicted intent are different, probability fusion is performed to reduce the prediction probability of the first predicted intent and the prediction probability of the second predicted intent, and the first predicted intent and the reduced prediction probability and the second predicted intent and the reduced prediction probability are used as the user's true intent recognition result.

[0028] To achieve the above and other related objectives, this application also provides a voice intent recognition device, comprising:

[0029] The data acquisition module is used to acquire the user's current voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data;

[0030] The first intent recognition module is used to perform user intent recognition on the user's current voice input text data using a multi-classification model, so as to obtain the first intent recognition result;

[0031] The second intent recognition module is used to perform user intent recognition based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environment data through a multimodal recognition model, so as to obtain the second intent recognition result.

[0032] The result fusion module is used to fuse the first intent recognition result with the second intent recognition result to obtain the user's true intent recognition result.

[0033] To achieve the above and other related objectives, this application also provides a vehicle control method, comprising:

[0034] Acquire the user's current voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data;

[0035] A multi-classification model is used to perform user intent recognition on the user's current voice input text data to obtain a first intent recognition result;

[0036] Using a multimodal recognition model, user intent is identified based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data, in order to obtain a second intent recognition result.

[0037] The first intent recognition result and the second intent recognition result are fused together to obtain the user's true intent recognition result;

[0038] Based on the user's true intent recognition results, vehicle control commands are generated to achieve vehicle cockpit control.

[0039] The vehicle cockpit voice intent recognition method, device, and vehicle control method of this application acquire user's current voice input text data, user's current facial expression and body language data, and vehicle internal and external environment data; use a multi-classification model to perform user intent recognition on the user's current voice input text data to obtain a first intent recognition result; use a multi-modal recognition model to perform user intent recognition based on the user's current voice input text data, user's historical voice input text data, user's current facial expression and body language data, and vehicle internal and external environment data to obtain a second intent recognition result; fuse the first intent recognition result and the second intent recognition result to obtain the user's true intent recognition result. By combining a multi-classification model and a multi-modal recognition model for user voice intent recognition, the accuracy of voice intent understanding can be improved, thereby enhancing the intelligence of the in-vehicle dialogue system and the user experience. Attached Figure Description

[0040] Figure 1 The diagram shown is a flowchart illustrating the vehicle cockpit voice intent recognition method of this application.

[0041] Figure 2 The flowchart shown is for step S40.

[0042] Figure 3 The diagram shown is a functional block diagram of the vehicle cockpit voice intent recognition device of this application.

[0043] Figure 4 The diagram shown is a flowchart of the vehicle control method of this application. Detailed Implementation

[0044] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application.

[0045] Please see Figure 1-4 It should be noted that the illustrations provided in this embodiment are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0046] To improve the accuracy of voice meaning understanding in the cockpit and enhance the intelligence and user experience of in-vehicle dialogue systems, embodiments of this application disclose a vehicle cockpit voice recognition method. Wherein, Figure 1 A flowchart illustrating a preferred embodiment of the vehicle cockpit voice recognition method of this application is shown.

[0047] Please see Figure 1 The vehicle cockpit voice recognition method includes the following steps:

[0048] Step S11: Obtain the user's current voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data;

[0049] Step S12: Use a multi-classification model to perform user intent recognition on the user's current voice input text data to obtain a first intent recognition result;

[0050] Step S13: Using a multimodal recognition model, user intent is recognized based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environment data, to obtain a second intent recognition result;

[0051] Step S14: Fuse the first intent recognition result with the second intent recognition result to obtain the user's true intent recognition result.

[0052] By combining multi-classification models and multi-modal recognition models for user voice intent recognition and fusion, the accuracy of voice intent understanding can be improved, thereby enhancing the intelligence of in-vehicle dialogue systems and the user experience.

[0053] It should be noted that, typically, the vehicle cockpit voice recognition method of this application can be executed jointly by the vehicle controller and the cloud, or entirely by the cloud. For example, a multi-classification model runs on the vehicle controller, while a multimodal recognition model runs on the cloud. The multi-classification model on the vehicle controller performs user intent recognition on the user's current voice input text data to obtain a first intent recognition result. The multimodal recognition model on the cloud performs user intent recognition based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environment data to obtain a second intent recognition result, which is then sent to the vehicle controller. The vehicle controller then fuses the first intent recognition result with the second intent recognition result to obtain the user's true intent recognition result. The vehicle controller includes a domain controller for the vehicle cockpit or a central controller within the vehicle cockpit.

[0054] Of course, in other embodiments, the vehicle cockpit voice recognition method of this application can also be executed by the cloud alone, or when the processing power of the vehicle controller is sufficient, the vehicle controller can also be the execution subject alone.

[0055] The technical solution of this application will be explained in detail below with specific application examples.

[0056] First, execute step S11: acquire the user's current voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environment data.

[0057] When the vehicle cabin receives a wake-up command, it will use the vehicle cabin's sensors or functional modules to obtain the user's current voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data. The wake-up command can be customized.

[0058] Specifically, when the vehicle cockpit receives a wake-up command, it first acquires the user's current voice input information through the microphone or other sound sensors in the vehicle cockpit. Then, it uses a voice recognition module to perform content recognition on the current voice input information to obtain the user's current voice input text data. The voice recognition module can be, for example, an ASR (Automatic Sound Recognition) voice recognition module. After acquiring the user's current voice input text data, the vehicle cockpit saves it locally or uploads it to the cloud as the user's historical voice input text data.

[0059] When the vehicle cockpit receives the wake-up command, it can use an image acquisition module, such as a camera, to acquire image data of the user's current facial expressions and gestures as the user's current facial and body data, and save it locally or upload it to the cloud.

[0060] When the vehicle cabin receives the wake-up command, the vehicle cabin can obtain at least one of the following as vehicle interior and exterior environmental data through the vehicle's internal and external sensors and the network: vehicle interior temperature data, vehicle interior air pressure data, vehicle exterior temperature data, and weather conditions. This data can be stored locally or uploaded to the cloud.

[0061] Next, step S12 is executed, using a multi-classification model to perform user intent recognition on the user's current voice input text data to obtain a first intent recognition result.

[0062] After obtaining the user's current voice input text data, features can be extracted from the data, and the extracted features are input into a multi-classification model for user intent prediction. The user intent with the highest prediction probability (the first predicted intent) and its corresponding prediction probability are taken as the first intent recognition result. The multi-classification model can be a general vehicle cockpit intent recognition model, such as a decision tree model or a deep learning model that has been pre-trained with a large amount of sample data.

[0063] Next, step S13 is executed, whereby user intent is recognized using a multimodal recognition model based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data, to obtain a second intent recognition result. The multimodal recognition model can be a model obtained by fine-tuning a pre-trained CogVLM model or GPT model using training data.

[0064] After acquiring multimodal data such as the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and vehicle internal and external environmental data, feature extraction can be performed on these multimodal data. The extracted features are then input into a multimodal recognition model to predict user intent. The user intent with the highest predicted probability (the second predicted intent) and its corresponding prediction probability are used as the second intent recognition result. The user's current voice input text data and the user's historical voice input text data represent multiple conversations with the user, which can serve as contextual information to help the multimodal recognition model better understand the user's intent.

[0065] It should be noted that steps S12 and S13 can be performed simultaneously or sequentially, and this embodiment does not limit this.

[0066] Finally, step S14 is executed to fuse the first intent recognition result with the second intent recognition result to obtain the user's true intent recognition result.

[0067] In this embodiment, since the multi-classification model is a unimodal model, it is faster at intent recognition than the multimodal recognition model. Furthermore, because the multimodal recognition model requires higher processing power and is typically deployed in the cloud, data transmission can affect its real-time performance. Therefore, after obtaining the first and second intent recognition results, they are fused based on the similarities and differences in the predicted intents and probabilities of the two results. The first intent recognition result is used as the primary indicator to obtain the user's true intent recognition result, thus ensuring the speed of intent recognition. Of course, in other embodiments, the second intent recognition result can also be used as the primary indicator.

[0068] In one specific embodiment, the first intent recognition result and the second intent recognition result are fused to obtain the user's true intent recognition result, specifically including:

[0069] When the prediction probability of the first predicted intent is greater than or equal to the first preset threshold, it means that the confidence level of the first predicted intent is relatively high, and the first predicted intent and the corresponding prediction probability can be used as the user's true intent recognition result.

[0070] When the prediction probability of the first predicted intent is less than the first preset threshold, and the prediction probability of the second predicted intent is greater than or equal to the second preset threshold, that is, when the confidence of the first predicted intent is low or very low, and the confidence of the second predicted intent is high, the second predicted intent and its corresponding prediction probability are taken as the result of the user's true intent identification.

[0071] When the prediction probability of the first predicted intent is less than the first preset threshold and the prediction probability of the second predicted intent is less than the second preset threshold, that is, when the confidence of the first and second predicted intents is relatively low and the first and second predicted intents are the same, probability fusion is performed to enhance the prediction probability of the first or second predicted intent. The first predicted intent and the enhanced prediction probability or the second predicted intent and the enhanced prediction probability are used as the user's true intent recognition result. The enhanced prediction probability can improve the confidence of the predicted intent, enabling the vehicle cabin to more accurately identify the user's intent.

[0072] When the prediction probability of the first predicted intent is less than a first preset threshold and the prediction probability of the second predicted intent is less than a second preset threshold, that is, when the confidence of the first and second predicted intents is relatively low and the first and second predicted intents are different, probability fusion is performed to reduce the prediction probability of the first and second predicted intents. The first predicted intent and the reduced prediction probability, along with the second predicted intent and the reduced prediction probability, are used as the user's true intent recognition result. By reducing the prediction probability, the confidence of the predicted intent can be lowered, avoiding incorrect recognition of user intent in the vehicle cabin, thereby improving the accuracy of user intent recognition in the vehicle cabin.

[0073] It should be noted that the first preset threshold and the second preset threshold here can be the same or different, and can be set according to the accuracy of the multi-classification model and the multi-modal recognition model after training. As an example, both can be set to 0.9.

[0074] In some embodiments, after obtaining the first intent recognition result, it can be further determined whether entity information needs to be supplemented based on the category of the first predicted intent of the first intent recognition result. When entity information needs to be supplemented, entity information corresponding to the first intent recognition result is extracted from the user's current voice input text data. The first intent recognition result, the entity information, and the second intent recognition result are then fused to obtain the user's true intent recognition result. As an example, when the user's current voice input text data is "adjust the temperature to 26 degrees", the user intent (first predicted intent) is identified as "adjust the vehicle air conditioning" through a multi-classification model. The user intent of "adjust the vehicle air conditioning" requires supplemented entity information, so the entity information "26 degrees" can be extracted from the user's current voice input text data. The extracted entity information can be used as supplementary information for the true intent and fused with the first intent recognition result and the second intent recognition result to obtain the recognition result, the entity information, and the second intent recognition result.

[0075] The following examples illustrate several application scenarios of vehicle cockpit voice intent recognition methods.

[0076] 1. When the vehicle cockpit receives a relatively accurate command from the user's voice input, a multi-classification model can be used to accurately identify the user's true intent. Then, if further information is needed to supplement the true intent, the entity information in the user's voice input can be analyzed to obtain the user's true intent information, such as:

[0077] 1) Voice input: Turn on the air conditioner; at this time, you only need to execute the command to turn on the air conditioner, and the default temperature will be used.

[0078] 2) Voice input: Adjust the temperature to 26 degrees; the intention is to adjust the car's air conditioning, the physical information is 26 degrees, and the actual intention is supplemented.

[0079] Second, when the vehicle cabin receives the current user's voice input but cannot accurately identify the user's true intention using a multi-classification model, it can understand the user's true purpose based on a multi-modal recognition model combined with contextual information. For example:

[0080] Sequential voice input information: "Open QQ Music" - "Play Seven Mile Fragrance" - "Pause for a while". Among them, "Open QQ Music" and "Play Seven Mile Fragrance" are the user's historical voice input text data, while "Pause for a while" is the user's current voice input text data. At this time, the multi-classification model cannot accurately identify the user's intention based on "Pause for a while". In this case, the multimodal recognition model can be used to understand the user's true intention to pause QQ Music playback by combining contextual information.

[0081] Third, leverage multimodal interaction and the external environment to understand intent. For example:

[0082] When the voice input is "It's a bit hot," the multi-classification model cannot accurately identify the user's intention. However, the image acquisition module detects that the user (car owner) is making a downward motion towards the car window, and / or the vehicle's sensors detect that the temperature inside the vehicle is higher than the temperature outside. Based on this multimodal data, the multimodal recognition model can identify that the user's intention is to open the car window.

[0083] IV. Integration enhances intent understanding.

[0084] Since both multi-classification models and multimodal recognition models predict user intent using probability values, the two parts can be merged to combine intent, entity recognition, and multimodal interaction recognition, thereby further improving the accuracy of understanding user intent.

[0085] like Figure 3 As shown, Figure 3 A functional block diagram of a preferred embodiment of the vehicle cockpit voice intent recognition device 11 of this application is shown. The vehicle cockpit voice intent recognition 11 includes a data acquisition module 111, a first intent recognition module 112, a second intent recognition module 113, and a result fusion module 114.

[0086] The system includes a data acquisition module 111, which acquires the user's current voice input text data, the user's current facial expression and body language data, and vehicle interior and exterior environment data; a first intent recognition module 112, which uses a multi-classification model to perform user intent recognition on the user's current voice input text data to obtain a first intent recognition result; a second intent recognition module 113, which uses a multi-modal recognition model to perform user intent recognition based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and vehicle interior and exterior environment data to obtain a second intent recognition result; and a result fusion module 114, which fuses the first intent recognition result with the second intent recognition result to obtain the user's true intent recognition result.

[0087] It should be noted that the vehicle cockpit voice intent recognition device 11 of this application is a system corresponding to the aforementioned vehicle cockpit voice intent recognition method. The functional modules in the vehicle cockpit voice intent recognition device 11 correspond to the corresponding steps in the vehicle cockpit voice intent recognition method. The vehicle cockpit voice intent recognition device 11 of this application can be implemented in conjunction with the vehicle cockpit voice intent recognition method. The relevant technical details mentioned in the vehicle cockpit voice intent recognition method of this application are still valid in the vehicle cockpit voice intent recognition device 11, and will not be repeated here to avoid repetition. Correspondingly, the relevant technical details mentioned in the vehicle cockpit voice intent recognition device 11 of this application can also be applied to the aforementioned vehicle cockpit voice intent recognition method.

[0088] It should be noted that the aforementioned functional modules can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. Additionally, these modules can be fully or partially integrated together, or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, some or all of the steps of the above method, or the aforementioned functional modules, can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0089] Figure 4 A flowchart illustrating a preferred embodiment of the vehicle control method of this application is shown. Please refer to... Figure 4 As shown, the vehicle control method includes the following steps:

[0090] Step S21: Obtain the user's current voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data;

[0091] Step S22: Use a multi-classification model to perform user intent recognition on the user's current voice input text data to obtain a first intent recognition result;

[0092] Step S23: Using a multimodal recognition model, user intent is recognized based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environment data to obtain a second intent recognition result;

[0093] Step S24: Fuse the first intent recognition result with the second intent recognition result to obtain the user's true intent recognition result;

[0094] Step S25: Generate vehicle control commands based on the user's true intent recognition results, and control the vehicle based on the vehicle control commands.

[0095] Among them, steps S21-S24 and Figure 1 The steps S11-S14 of the vehicle cockpit voice intent recognition method shown are the same, as detailed in the relevant sections above, and will not be repeated here.

[0096] In step S25, after obtaining the user's true intent recognition result through fusion, the system can understand the user's true intent based on the recognition result and issue control commands to the vehicle cabin accordingly. The vehicle cabin then performs corresponding operations based on the control commands, such as lowering / raising the temperature, turning the seat massage on / off, opening / closing the target window, raising / lowering the vehicle's volume, and adjusting the seat angle forward / backward. After the vehicle cabin performs the corresponding operations based on the control commands, it can also synthesize a response voice through the vehicle cabin's voice synthesis module and play it through the vehicle cabin's speakers, thus realizing human-computer interaction.

[0097] The user's true intent recognition result includes at least one user intent and its corresponding predicted probability. Before generating vehicle control commands based on the user's true intent recognition result, it is necessary to first determine the predicted probability (here referring to the fused predicted probability) corresponding to the user intent to judge the confidence level of the user intent. When the predicted probability is greater than a preset probability threshold (which may be the first preset threshold, the second preset threshold, or other suitable preset threshold mentioned above), the user intent is considered to have high confidence, and control commands can be directly generated based on the user intent. When the predicted probability of the user intent is less than the preset probability threshold, the user intent is considered to have low confidence. In this case, an inquiry statement can be synthesized by the speech synthesis module and played through the speakers in the vehicle cabin to further interact with the user. Based on the results of the user's further interaction, the user's true intent is re-determined, and vehicle control commands are generated based on the re-determined user's true intent.

[0098] In summary, the vehicle cockpit voice intent recognition method, device, and vehicle control method of this application acquire user's current voice input text data, user's current facial expression and body language data, and vehicle internal and external environment data; utilize a multi-classification model to perform user intent recognition on the user's current voice input text data to obtain a first intent recognition result; utilize a multi-modal recognition model to perform user intent recognition based on the user's current voice input text data, user's historical voice input text data, user's current facial expression and body language data, and vehicle internal and external environment data to obtain a second intent recognition result; and fuse the first intent recognition result and the second intent recognition result to obtain the user's true intent recognition result. By combining a multi-classification model and a multi-modal recognition model for user voice intent recognition, the accuracy of voice intent understanding can be improved, thereby enhancing the intelligence of the in-vehicle dialogue system and the user experience.

[0099] Throughout this description, numerous specific details, such as examples of components and / or methods, are provided to provide a complete understanding of the embodiments of this application. However, those skilled in the art will recognize that the embodiments of this application may be practiced without one or more of these specific details or by other devices, systems, components, methods, parts, materials, components, etc. In other instances, well-known structures, materials, or operations have not been specifically shown or described in detail to avoid obscuring aspects of the embodiments of this application.

[0100] The above description of the embodiments shown in this application (including the content set forth in the abstract of the specification) is not intended to be an exhaustive enumeration or to limit this application to the precise forms disclosed herein. Although specific embodiments and examples of this application have been described herein for illustrative purposes only, various equivalent modifications are possible within the spirit and scope of the invention, as will be recognized and understood by those skilled in the art. As indicated, these modifications can be made to this application in accordance with the above description of the embodiments described herein, and such modifications will be within the spirit and scope of this application.

Claims

1. A method for recognizing voice intent in a vehicle cockpit, characterized in that, include: Acquire the user's current voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data; A multi-classification model is used to perform user intent recognition on the user's current voice input text data to obtain a first intent recognition result; Using a multimodal recognition model, user intent is identified based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data, in order to obtain a second intent recognition result. The first intent recognition result and the second intent recognition result are fused together to obtain the user's true intent recognition result; The method of fusing the first intent recognition result and the second intent recognition result to obtain the user's true intent recognition result includes: fusing based on the similarities and differences and prediction probabilities of the predicted intents in the first intent recognition result and the second intent recognition result, and using the first intent recognition result as the primary factor to obtain the user's true intent recognition result; when the prediction probability of the first predicted intent is less than a first preset threshold, the prediction probability of the second predicted intent is less than a second preset threshold, and the first predicted intent and the second predicted intent are the same, probability fusion is performed to enhance the prediction probability of the first predicted intent or the second predicted intent, and the first predicted intent and the enhanced prediction probability or the second predicted intent and the enhanced prediction probability are used as the user's true intent recognition result; when the prediction probability of the first predicted intent is less than the first preset threshold, the prediction probability of the second predicted intent is less than the second preset threshold, and the first predicted intent and the second predicted intent are different, probability fusion is performed to reduce the prediction probability of the first predicted intent and the second predicted intent, and the first predicted intent and the reduced prediction probability, and the second predicted intent and the reduced prediction probability are used as the user's true intent recognition result.

2. The vehicle cockpit voice intent recognition method according to claim 1, characterized in that, Also includes: Save the user's current voice input text data as the user's historical voice input text data.

3. The vehicle cockpit voice intent recognition method according to claim 1, characterized in that, The method of fusing the first intent recognition result with the second intent recognition result to obtain the user's true intent also includes: Determine whether additional entity information is needed based on the intent category of the first intent recognition result; When it is necessary to supplement entity information, entity information corresponding to the first intent recognition result is extracted from the user's current voice input text data. The first intent recognition result, the entity information, and the second intent recognition result are fused together to obtain the user's true intent recognition result.

4. The vehicle cockpit voice intent recognition method according to claim 1, characterized in that, Obtain the user's current voice input text data, including: Obtain the user's current voice input information; The speech recognition module performs content recognition on the current speech input information to obtain the user's current speech input text data.

5. The vehicle cockpit voice intent recognition method according to claim 1, characterized in that, The vehicle's internal and external environmental data includes at least one of the following: vehicle interior temperature data, vehicle interior air pressure data, vehicle exterior temperature data, and weather conditions.

6. The vehicle cockpit voice intent recognition method according to claim 1, characterized in that, The multi-classification model includes a decision tree model or a deep learning model; the multimodal recognition model includes a CogVLM model or a GPT model.

7. The vehicle cockpit voice intent recognition method according to claim 1, characterized in that, The multimodal recognition model is set up in the cloud.

8. The vehicle cockpit voice intent recognition method according to claim 1, characterized in that, The first intent recognition result and the second intent recognition result are fused to obtain the user's true intent recognition result, including: When the predicted probability of the first predicted intent is greater than or equal to the first preset threshold, the first predicted intent and the corresponding predicted probability are taken as the user's true intent recognition result. When the prediction probability of the first predicted intent is less than the first preset threshold, while the prediction probability of the second predicted intent is greater than or equal to the second preset threshold, the second predicted intent and its corresponding prediction probability are taken as the result of the user's true intent identification.

9. A voice intent recognition device, characterized in that, include: The data acquisition module is used to acquire the user's current voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data; The first intent recognition module is used to perform user intent recognition on the user's current voice input text data using a multi-classification model, so as to obtain the first intent recognition result; The second intent recognition module is used to perform user intent recognition based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environment data through a multimodal recognition model, so as to obtain the second intent recognition result. The result fusion module is used to fuse the first intent recognition result with the second intent recognition result to obtain the user's true intent recognition result; The method of fusing the first intent recognition result and the second intent recognition result to obtain the user's true intent recognition result includes: fusing based on the similarities and differences and prediction probabilities of the predicted intents in the first intent recognition result and the second intent recognition result, and using the first intent recognition result as the primary factor to obtain the user's true intent recognition result; when the prediction probability of the first predicted intent is less than a first preset threshold, the prediction probability of the second predicted intent is less than a second preset threshold, and the first predicted intent and the second predicted intent are the same, probability fusion is performed to enhance the prediction probability of the first predicted intent or the second predicted intent, and the first predicted intent and the enhanced prediction probability or the second predicted intent and the enhanced prediction probability are used as the user's true intent recognition result; when the prediction probability of the first predicted intent is less than the first preset threshold, the prediction probability of the second predicted intent is less than the second preset threshold, and the first predicted intent and the second predicted intent are different, probability fusion is performed to reduce the prediction probability of the first predicted intent and the second predicted intent, and the first predicted intent and the reduced prediction probability, and the second predicted intent and the reduced prediction probability are used as the user's true intent recognition result.

10. A vehicle control method, characterized in that, include: Acquire the user's current voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data; A multi-classification model is used to perform user intent recognition on the user's current voice input text data to obtain a first intent recognition result; Using a multimodal recognition model, user intent is identified based on the user's current voice input text data, the user's historical voice input text data, the user's current facial expression and body language data, and the vehicle's internal and external environmental data, in order to obtain a second intent recognition result. The first intent recognition result and the second intent recognition result are fused together to obtain the user's true intent recognition result; Based on the user's true intent recognition results, vehicle control commands are generated to achieve vehicle cockpit control; The method of fusing the first intent recognition result and the second intent recognition result to obtain the user's true intent recognition result includes: fusing based on the similarities and differences and prediction probabilities of the predicted intents in the first intent recognition result and the second intent recognition result, and using the first intent recognition result as the primary factor to obtain the user's true intent recognition result; when the prediction probability of the first predicted intent is less than a first preset threshold, the prediction probability of the second predicted intent is less than a second preset threshold, and the first predicted intent and the second predicted intent are the same, probability fusion is performed to enhance the prediction probability of the first predicted intent or the second predicted intent, and the first predicted intent and the enhanced prediction probability or the second predicted intent and the enhanced prediction probability are used as the user's true intent recognition result; when the prediction probability of the first predicted intent is less than the first preset threshold, the prediction probability of the second predicted intent is less than the second preset threshold, and the first predicted intent and the second predicted intent are different, probability fusion is performed to reduce the prediction probability of the first predicted intent and the second predicted intent, and the first predicted intent and the reduced prediction probability, and the second predicted intent and the reduced prediction probability are used as the user's true intent recognition result.

Images