A vehicle-mounted dialogue recognition method and system based on multi-mode fusion

Abstract

This application relates to a vehicle-mounted dialogue recognition method and system based on multimodal fusion. Through an information acquisition system, it collects voice and image information, combines voice, text, and image information, aligns the relevant information, and uses a large-scale model technique to more accurately determine the user's dialogue intent. Specifically, it parses voice and image information, generates feature values, and aligns the voice and image information using timestamps. The aligned voice and image information features are then used in a three-level arbitration mechanism—simple rules, simplified models, and deep models—for the final decision. The multimodal method provides richer information sources, enabling more accurate determination of the user's dialogue intent. By combining voice, text, image information, and vehicle status information, and aligning time information, the system exhibits better adaptability and robustness, accurately determining the user's intent in complex environments. The multi-level arbitration design balances response speed and recognition effectiveness.

Description

Technical Field

[0001] The present application relates to the field of data model technology, and in particular to a method and system for in-vehicle dialogue recognition based on multi-mode fusion. Background Art

[0002] With the widespread application of artificial intelligence technology, how to judge whether the user is talking to the machine has become a difficult point in the application of artificial intelligence technology. There are two main reasons for this. On the one hand, it is not convenient for users to conduct human-computer information interaction through a single mode, that is, through text or through voice. On the other hand, in some cases, this single-mode method may not accurately capture the true behavior and intention of the person who issued the instruction, especially in complex environments, such as the background sound of multiple people talking inside the vehicle will interfere with the machine's voice information recognition and information extraction. To this end, it is necessary to propose a vehicle-mounted dialogue recognition method and system based on multi-mode fusion to address the defect of inaccurate judgment results of user-machine dialogue behavior in the traditional single-mode human-computer information interaction method. Summary of the invention

[0003] Based on this, it is necessary to propose an in-vehicle dialogue recognition method and system based on multi-modal fusion to address the defect of inaccurate judgment results of user-machine dialogue behavior in the traditional single-modal human-computer information interaction method.

[0004] The present application provides a vehicle-mounted dialogue recognition method based on multi-mode fusion, comprising:

[0005] Based on the information collection system, voice information and image information are collected;

[0006] Analyze voice information and image information to generate feature values;

[0007] Assign a timestamp to each feature value;

[0008] Receive all feature values;

[0009] Select a timestamp;

[0010] Incorporate all feature values ​​with the same timestamp into the multimodal large model;

[0011] Obtain the probability distribution of dialogue behaviors;

[0012] Based on the probability distribution of the dialogue behavior, the program represented by the feature value is executed;

[0013] Return to selecting a timestamp until all timestamps are selected.

[0014] The present application also provides a vehicle-mounted dialogue recognition system based on multi-mode fusion, comprising:

[0015] A server, used to execute the above-mentioned vehicle-mounted dialogue recognition method based on multi-mode fusion;

[0016] Voice input devices are all connected to the server in communication;

[0017] The cameras are all connected to the server for communication.

[0018] The present application relates to a method and system for in-vehicle conversation recognition based on multi-modal fusion. Through an information collection system, voice information and image information are collected, and after combining voice, text and image information, the relevant information is aligned, and the technology based on the large model is used to more accurately judge the user's conversation intention. Specifically, the voice information and image information are parsed, feature values ​​are generated, and the voice information and image information are aligned using timestamps. The aligned voice information and image information features are used for final decision through a three-level arbitration mechanism, simple rules, simplified models, and deep models. The multimodal method provides a richer source of information and can more accurately judge the user's conversation intention. After combining voice, text, image information and vehicle status information and aligning time information, the system has better adaptability and robustness, and can accurately judge the recognition intention in a complex environment. Through the design of multi-level arbitration, both response speed and recognition effect can be taken into account. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The illustrative embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application.

[0020] FIG1 is a method flow chart of a method for in-vehicle dialogue recognition based on multi-modal fusion provided in one embodiment of the present application.

[0021] Figure 2 A schematic diagram of the module structure of an in-vehicle dialogue recognition system based on multi-mode fusion provided in one embodiment of the present application.

[0022] FIG3 is a schematic diagram of the structure of an in-vehicle dialogue recognition system based on multi-mode fusion provided in one embodiment of the present application.

[0023] Reference numerals:

[0024] 100-server; 200-voice input device; 300-camera. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] The present application provides a vehicle-mounted dialogue recognition method based on multi-mode fusion.

[0027] As shown in FIG1 , in one embodiment of the present application, a vehicle-mounted dialogue recognition method based on multi-mode fusion includes:

[0028] S100, based on the information collection system, collects voice information and image information.

[0029] Specifically, the information collection system actually collects three types of information, including text information, voice information and image information.

[0030] This information enables the collection of information on a variety of possible interaction scenarios in complex in-vehicle environments, such as making phone calls, meetings, and passenger chats.

[0031] The voice information includes one or more of vehicle status audio and human speech audio.

[0032] S200, analyzing the voice information and the image information to generate feature values.

[0033] Specifically, text information, voice information and image information can all be converted into feature values. Since text information is relatively more direct and clear, this embodiment takes voice information and image information, which are more difficult to convert, as examples.

[0034] S300, assigning a timestamp to each feature value.

[0035] Specifically, the timestamp is a good reference system for aligning the voice information and image information in the space. The voice information and image information, at the same timestamp, can be combined to comprehensively judge whether the dialogue behavior is a voice command.

[0036] S400: Receive all characteristic values.

[0037] S500, select a timestamp.

[0038] Specifically, this step is actually the alignment process of feature values, which is completed by the data alignment processing module.

[0039] S600, incorporate all feature values ​​with the same timestamp into the multimodal large model.

[0040] Specifically, the multimodal large model includes a first-level arbitrator.

[0041] Alternatively, the multimodal large model includes a first-level arbitrator and a second-level arbitrator.

[0042] Alternatively, the multimodal large model includes a first-level arbitrator, a second-level arbitrator and a third-level arbitrator.

[0043] S700: Obtain the occurrence probability distribution of the dialogue behavior.

[0044] S800, executing a program represented by the feature value based on the probability distribution of the dialogue behavior.

[0045] S900, returning to the step of selecting a timestamp until all timestamps are selected.

[0046] More specifically, the vehicle status audio includes the wind resistance audio, tire noise audio, in-vehicle multimedia audio, etc. during vehicle driving.

[0047] Human voice audio includes phone call audio, meeting audio, passenger chat audio, etc.

[0048] The present application relates to a method for in-vehicle conversation recognition based on multi-modal fusion. Through an information collection system, voice information and image information are collected, and after combining voice, text and image information, the relevant information is aligned, and the technology based on the large model is used to more accurately judge the user's conversation intention. Specifically, the voice information and image information are parsed, feature values ​​are generated, and the voice information and image information are aligned using timestamps. The aligned voice information and image information features are used for final decision making through a three-level arbitration mechanism, simple rules, simplified models, and deep models. The multimodal method provides a richer source of information and can more accurately judge the user's conversation intention. After combining voice, text, image information and vehicle status information and aligning time information, the system has better adaptability and robustness, and can accurately judge the recognition intention in a complex environment. Through the design of multi-level arbitration, both response speed and recognition effect can be taken into account.

[0049] In one embodiment of the present application, S100 includes:

[0050] S110, starting a voice input device.

[0051] S120, receiving real-time voice waveform data.

[0052] S130, converting the speech waveform into text based on the speech recognition model.

[0053] S140, analyzing the speech waveform.

[0054] S150, extracting timbre data, frequency data, speech speed, volume and tone intensity marker information.

[0055] Specifically, vehicle status such as vehicle speed, number of passengers, in-car multimedia volume, steering wheel status, automatic driving mode, passenger status, window opening status, geographic location, road conditions, etc. will also affect human-computer voice interaction.

[0056] S160, incorporating the extracted sign information into the text.

[0057] S170, start the camera.

[0058] S180, obtaining one or more types of image information in a picture, continuous picture frames, and a video.

[0059] S190, extracting dynamic information of the number of people, facial movements, and gestures in the image information based on the image information.

[0060] Specifically, it is used to obtain image information from the current camera in the car. The camera may include one or more cameras to capture visual information in the car, such as the number of passengers, the speaker's facial expressions, gestures, phone calls, chat status or other relevant information.

[0061] In one embodiment of the present application, S200 includes:

[0062] S211, calling the sound spectrum data, wherein the sound spectrum data includes timbre data, frequency data, speech speed, volume and tone intensity.

[0063] S212, using the frequency data to determine the type of speech utterance in the text.

[0064] S213, extracting human voice data from the voice emitting object category.

[0065] Specifically, by capturing the user's voice waveform signal, the user's voice input can be obtained in real time, such as the passenger's voice speed, volume, tone, pauses or other relevant information.

[0066] More specifically, the audio information of each speaker is distinguished through timbre data, frequency data, speech speed, volume and tone intensity, which is helpful for aligning the speaker with the action in the image.

[0067] S214, distinguishing the human voice data of each speaker in the human voice data based on the timbre data, the speed, volume and tone strength of the voice.

[0068] S215, selecting voice data of a speaker.

[0069] S216, based on the dynamic information of all facial movements, determine the dynamic information of the facial movements corresponding to the voice data of the person.

[0070] S217, forming a mapping relationship between the person's voice data and the dynamic information of each matching facial action.

[0071] S218, returning to the process of selecting the voice data of a speaker until the voice data of all speakers are selected.

[0072] Specifically, through timestamps, image information and audio information are aligned and fused to generate aligned features. After combining voice, text, image information and vehicle status information and aligning time information, the system has better adaptability and robustness, and can accurately judge the intention of data recognition in complex environments.

[0073] In one embodiment of the present application, S200 further includes:

[0074] S221, select a text.

[0075] S222: Based on a text encoder, convert the text into a language feature value.

[0076] S223, returning to the process of selecting a text until all texts are selected.

[0077] S224, select an image information.

[0078] S225, based on the audio encoder, extracts the voice waveform data in the image information in real time.

[0079] S226, returning the method of converting the speech waveform into text based on the speech recognition model to generate language feature values ​​about the image information.

[0080] S227, based on the image encoder, convert the dynamic information into image feature values.

[0081] S228, returning to the step of selecting an image information until all the image information is selected.

[0082] Specifically, the audio encoder uses the Wav2Vec model to convert the original audio waveform or spectrogram into a feature representation. The text encoder uses BERT or other Transformer models to convert text sequences into feature representations. The image encoder uses a 3D convolutional neural network (such as I3D, C3D) or a convolution-Transformer hybrid model to convert video frame sequences into feature representations. The vehicle information encoder can encode the characteristic values ​​of the vehicle status. The vehicle information encoder converts the vehicle status information into a feature vector according to different categories through standardization, normalization, time series coding, etc.

[0083] The output feature representation of the encoder can be expressed as:

[0084]

[0085] Among them, I, A, T and V are the feature representations of video, audio, text and vehicle status respectively.

[0086] In one embodiment of the present application, S600 includes:

[0087] S611, receiving language feature values.

[0088] S612, calling the large language model.

[0089] S613, inputting the language feature values ​​into the large language model.

[0090] S614: Output the probability distribution of the dialogue intention according to the language feature value.

[0091] S615, receiving image feature values.

[0092] S616, calling the image processing model.

[0093] S617, inputting the image feature values ​​into the image processing model.

[0094] S618: Output the probability distribution of the behavior intention according to the image feature value.

[0095] Specifically, the features of each modality are directly concatenated into a long vector and then input into a simple classifier such as a shallow neural network, and then the features of each modality in video, audio, text, and vehicle status are weighted averaged.

[0096] It is worth mentioning that this embodiment is used to demonstrate the deep arbitration model, so the modal characteristics of the vehicle state can be used as a reference for the judgment of the multi-head self-attention mechanism and the feedforward neural network.

[0097] More specifically,

[0098] in is the weight of each modal feature.

[0099] is the weighted average function.

[0100] A shared Transformer encoder is used to process the feature representations of different modalities to achieve multimodal alignment. The input of the Transformer encoder is the feature representation of each modality, which is processed through a multi-head self-attention mechanism and a feed-forward neural network.

[0101] The calculation of the Transformer encoder can be expressed as:

[0102] F(H)=TransformerEncoder(F)

[0103] Among them, F is the concatenation or weighted combination of all modal feature representations, and H is the output of the Transformer encoder.

[0104] In one embodiment of the present application, S600 further includes:

[0105] S621, generating a long vector based on the characteristic value.

[0106] S622, performing weighted averaging on each type of characteristic value of the long vector.

[0107] S623, using an encoder to encode the long vector after weighted averaging to generate a judgment function.

[0108] S624, calling the judgment function and the simple rule to generate a fast regularized recognition function; the fast regularized recognition function is the judgment rule of the first-level arbitrator.

[0109] S625, receiving characteristic values.

[0110] S626, generating a first-level arbitrator output result based on the fast regularization recognition function.

[0111] When the multimodal large model includes a first-level arbitrator, the output result of the first-level arbitrator is the vehicle-mounted dialogue recognition result. If the output result of the first-level arbitrator is established, no human-machine communication dialogue occurs.

[0112] If the output result of the first-level arbitrator is not valid, a human-computer communication dialogue will occur.

[0113] Specifically, the first-level arbitrator: simple rules, that is, the rule model can be defined as:

[0114] Fast regularized recognition:

[0115] P1(intent)=P(PhoneCall)∨P(Conversation)∨P(QuietDriving)∨P(ExternalNoise) (As long as any intention is established, the system will recognize it)

[0116] Phone_call: Determine whether there is a phone call based on the video signal.

[0117] Conversations: Conversations between multiple people in the car.

[0118] External Noise: Sound outside the vehicle.

[0119] Quiet Driving: Determine if the user is driving quietly based on visual signals.

[0120] The second-level arbitrator uses a simple feature fusion method to perform intent recognition through a machine learning model.

[0121] The simplified model can be defined as:

[0122] P2(intent)=SecondLevelArbitration(F(simple)).

[0123] In one embodiment of the present application, S600 further includes:

[0124] S631, calling the machine learning model.

[0125] S632, based on the feature fusion function encoder and judgment function of the second-level arbitrator, generate the input code of the machine learning model.

[0126] S633, input the input code of the machine learning model into the machine learning model to generate the output result of the second-level arbitrator.

[0127] When the multimodal large model includes a second-level arbitrator, the output result of the second-level arbitrator is the vehicle-mounted dialogue recognition result. If the output result of the second-level arbitrator is established, no human-machine communication dialogue occurs.

[0128] If the output result of the second-level arbitrator is not valid, a human-computer communication dialogue will occur.

[0129] Specifically, the third-level arbitrator: deep arbitration model uses a complex multimodal fusion method to perform advanced feature extraction and intent recognition through a deep learning model.

[0130] P3(intent)=ThirdLevelArbitration(F(deep))

[0131] The first-level, second-level, and third-level arbitrators can be integrated together to form a multi-level arbitrator process.

[0132] P1(intent), P2(intent) and P3(intent) are the probability distributions of the first-level, second-level and third-level arbitrators, respectively.

[0133] like Figure 2As shown in the figure, the final judgment result of the arbitration module is fed back to the intent judgment module so that the system can respond to the user's intention. The arbitration result is input into the intent judgment module through an interface or a message passing mechanism. The final dialogue intention judgment result. Through the multi-model fusion architecture, a comprehensive judgment is made based on the output of each independent model, providing a more robust and accurate dialogue intention judgment. The introduction of the arbitration module ensures the scientificity and reliability of the comprehensive judgment, and the final feedback mechanism ensures the system's timely response to the user's intention. Through such a design, this module can effectively deal with the lack and uncertainty of multiple modal information and improve the overall performance and user experience of the system.

[0134] In one embodiment of the present application, S600 further includes:

[0135] S641, calling the deep learning model.

[0136] S642, based on the feature fusion function encoder and judgment function of the third-level arbitrator, generate the input code of the deep learning model.

[0137] S643, input the input code of the deep learning model into the machine learning model to generate the output result of the third-level arbitrator.

[0138] S644, when the multimodal large model includes a first-level arbitrator, a second-level arbitrator, and a third-level arbitrator, if the output result of the first-level arbitrator is valid, the output result of the first-level arbitrator is output.

[0139] S645, if the output result of the first-level arbitrator is invalid and the output result of the second-level arbitrator is valid, then output the output result of the second-level arbitrator.

[0140] S646, if the output result of the first-level arbitrator is invalid, the second-level arbitrator is invalid, and the third-level arbitrator is valid, then output the output result of the third-level arbitrator.

[0141] S647, if the output result of the first-level arbitrator is invalid, the second-level arbitrator is invalid, and the third-level arbitrator is invalid, it is determined that a human-computer interaction situation occurs.

[0142] Specifically, it is determined whether the output result of the first-level arbitrator is valid.

[0143] If the output result of the first-level arbitrator is valid, the output result of the first-level arbitrator is output.

[0144] If the output result of the first-level arbitrator is not valid, it is determined whether the output result of the second-level arbitrator is valid.

[0145] If the output result of the second-level arbitrator is valid, the output result of the second-level arbitrator is output.

[0146] If the output result of the second-level arbitrator is not valid, it is determined whether the output result of the third-level arbitrator is valid.

[0147] If the output result of the third-level arbitrator is valid, the output result of the third-level arbitrator is output.

[0148] If the output result of the third-level arbitrator is not valid, it is determined that a human-computer communication dialogue situation occurs.

[0149] The present application provides an in-vehicle dialogue recognition system based on multi-mode fusion.

[0150] As shown in FIG. 3 , in one embodiment of the present application, a vehicle-mounted dialogue recognition system based on multi-modal fusion includes a server 100 , a voice input device 200 , and a camera 300 .

[0151] The server 100 is used to execute the in-vehicle conversation recognition method based on multi-modal fusion.

[0152] The voice input devices 200 are all connected to the server 100 for communication.

[0153] The cameras 300 are all connected to the server 100 for communication.

[0154] The server 100 includes a speech waveform capture module, a speech-to-text module, an image acquisition module, an intention judgment module, a multimodal large model module, an intention execution processing module, and a recognition result return module. The speech waveform capture module, the speech-to-text module, and the image acquisition module are all in communication connection with the intention judgment module. The multimodal large model module is in communication with the intention judgment module. The intention execution processing module is in communication with the intention judgment module. The recognition result return module is in communication with the intention judgment module.

[0155] The present application relates to a vehicle-mounted dialogue recognition system based on multi-modal fusion. The server 100 collects voice information and image information through information collection systems such as voice input devices 200, text input devices and cameras 300, combines voice, text and image information, aligns the relevant information, and uses large model technology to more accurately judge the user's dialogue intention. Specifically, the voice information and image information are parsed, feature values ​​are generated, and the voice information and image information are aligned using timestamps. The aligned voice information and image information features are used for final decision-making through a three-level arbitration mechanism, simple rules, simplified models, and deep models. The multimodal method provides a richer source of information and can more accurately judge the user's dialogue intention. After combining voice, text, image information and vehicle status information and aligning time information, the system has better adaptability and robustness, and can accurately judge the recognition intention in a complex environment. Through the design of multi-level arbitration, both response speed and recognition effect can be taken into account.

[0156] The technical features of the above-described embodiments can be combined arbitrarily, and the execution order of each method step is not limited. In order to make the description concise, all possible combinations of the technical features in the above-described embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-described embodiments only express several implementation methods of the present application, and the description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the patent of this application. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several deformations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be based on the attached claims.

Claims

1. A vehicle-mounted dialogue recognition method based on multi-mode fusion, characterized in that: include: Based on the information collection system, collect vehicle status, voice information and image information; The voice information includes one or more of vehicle audio and human speech audio; Analyze vehicle status, voice information and image information to generate feature values; Assign a timestamp to each feature value; Receive all feature values; Select a timestamp; Incorporating all feature values ​​with the same timestamp into a multimodal large model; the multimodal large model includes a first-level arbitrator or the multimodal large model includes a second-level arbitrator or the multimodal large model includes a first-level arbitrator, a second-level arbitrator and a third-level arbitrator; Obtain the probability distribution of dialogue behaviors; Based on the probability distribution of the dialogue behavior, the program represented by the feature value is executed; Return to selecting a timestamp until all timestamps are selected; The method of collecting voice information and image information based on the information collection system includes: starting a voice input device; Receive real-time data of voice waveform; Convert speech waveforms into text based on speech recognition models; Analyze speech waveforms; Extracting timbre data, frequency data, speech speed, volume and tone intensity marker information; Incorporating the extracted sign information into the text; Start the camera; Acquire one or more image information in pictures, continuous picture frames and videos; Based on the image information, dynamic information such as the number of people, facial movements and gestures in the image information is extracted.

2. The in-vehicle dialogue recognition method based on multi-mode fusion according to claim 1 is characterized in that: Parsing the voice information and the image information to generate feature values ​​includes: calling the sound spectrum data; the sound spectrum data includes timbre data, frequency data, speed of the voice, volume and tone intensity; Using frequency data to determine the types of speech artifacts in text; Extracting human voice data from the voice emitting object category; Distinguish the human voice data of each speaker in the human voice data based on the timbre data, the speed, volume and tone strength of the voice; Select a person's voice data of a speaking object; Determine the dynamic information of the facial action corresponding to the voice data of the person based on the dynamic information of all facial actions; Mapping the person's voice data to the dynamic information of each matching facial action; The voice data of a person who selects a speaker is returned until the voice data of all speakers are selected.

3. The vehicle-mounted dialogue recognition method based on multi-mode fusion according to claim 2 is characterized in that: The said analyzing the voice information and the image information and generating the feature value also includes: selecting a text; Based on the text encoder, the text is converted into a language feature value; Return to selecting a text until all texts are selected; Select an image message; Based on the audio encoder, the voice waveform data in the image information is extracted in real time; Returning the method of converting speech waveform into text based on the speech recognition model to generate language feature values ​​about image information; Based on the image encoder, the dynamic information is converted into image feature values; Return to the step of selecting an image information until all the image information is selected.

4. The vehicle-mounted dialogue recognition method based on multi-mode fusion according to claim 3 is characterized in that: The incorporating all feature values ​​having the same timestamp into the multimodal macro model includes: receiving language feature values; Calling a large language model; Input the language feature values ​​into the large language model; Output the probability distribution of conversation intention according to the language feature value; Receiving image feature values; Call the image processing model; Inputting image feature values ​​into the image processing model; According to the image feature values, the probability distribution of behavioral intention is output.

5. The in-vehicle dialogue recognition method based on multi-mode fusion according to claim 4 is characterized in that: The incorporating all feature values ​​having the same timestamp into the multimodal large model further includes: generating a long vector based on the feature values; Take a weighted average of each type of eigenvalue of a long vector; The encoder is used to encode the long vector after weighted averaging to generate a judgment function; Calling the judgment function and simple rules to generate a fast regularized recognition function; the fast regularized recognition function is the judgment rule of the first-level arbitrator; receiving characteristic values; Based on the fast regularized recognition function, a first-level arbitrator output result is generated; when the multimodal large model includes a first-level arbitrator, the first-level arbitrator output result is an in-vehicle dialogue recognition result.

6. The in-vehicle dialogue recognition method based on multi-mode fusion according to claim 5 is characterized in that: The incorporating all feature values ​​having the same timestamp into the multimodal large model further includes: calling a machine learning model; Generate input encoding of machine learning model based on feature fusion function encoder and judgment function of the second-level arbitrator; The input code of the machine learning model is input into the machine learning model to generate a second-level arbitrator output result; when the multimodal large model includes a second-level arbitrator, the second-level arbitrator output result is an in-vehicle dialogue recognition result.

7. The in-vehicle dialogue recognition method based on multi-mode fusion according to claim 6 is characterized in that: The incorporating all feature values ​​having the same timestamp into the multimodal large model further includes: calling a deep learning model; Generate input code of deep learning model based on feature fusion function encoder and judgment function of the third-level arbitrator; The input encoding of the deep learning model is fed into the machine learning model to generate the output result of the third-level arbitrator; When the multimodal large model includes a first-level arbitrator, a second-level arbitrator, and a third-level arbitrator, if the output result of the first-level arbitrator is valid, the output result of the first-level arbitrator is output; If the output result of the first-level arbitrator is invalid and the output result of the second-level arbitrator is valid, then the output result of the second-level arbitrator is output; If the output result of the first-level arbitrator is invalid, the second-level arbitrator is invalid, and the third-level arbitrator is valid, then the output result of the third-level arbitrator is output; If the output result of the first-level arbitrator is invalid, the second-level arbitrator is invalid, and the third-level arbitrator is invalid, it is determined that a human-computer interaction situation occurs.

8. A vehicle-mounted dialogue recognition system based on multi-mode fusion, characterized in that: include: A server, configured to execute the in-vehicle conversation recognition method based on multi-modal fusion as claimed in any one of claims 1 to 7; A voice input device, connected to the server for communication; A camera is connected to the server for communication.

9. The in-vehicle dialogue recognition system based on multi-mode fusion according to claim 8 is characterized in that: The server includes a speech waveform capture module, a speech-to-text module, an image acquisition module, an intention judgment module, a multimodal large model module, an intention execution processing module and a recognition result return module; The speech waveform capturing module, the speech-to-text conversion module and the image acquisition module are all in communication connection with the intention determination module; The multimodal large model module is communicatively connected with the intention judgment module; The intention execution processing module is communicatively connected with the intention judgment module; The recognition result returning module is communicatively connected with the intention judging module.

Images