Multi-modal interaction method and apparatus, controller, system, automobile, and storage medium

Abstract

A multi-modal interaction method and apparatus, a controller, a system, an automobile, and a storage medium. The method comprises: determining a current interaction script on the basis of a current interaction speech at an interaction moment; on the basis of a current interaction interface corresponding to the interaction moment, determining a current scene image and current scene data corresponding to the current interaction interface; using a multi-modal recognition model to perform multi-modal recognition on the current interaction script, the current scene image and the current scene data to determine a target control instruction; and executing the target control instruction to complete a human-computer interaction operation. The method can ensure the output efficiency and accuracy of target control instructions, and reduce the complex control logic such as customized templates, rules and associations in the development process, improving the adaptability and generalization ability of speech interaction.

Description

Multimodal interaction method, device, controller, system, automobile and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 28, 2023, with application number 202311294515.2 and application name “Multimodal interaction method, device, controller, system, automobile and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of human-computer interaction technology, and in particular to a multimodal interaction method, device, controller, system, automobile, and storage medium. Background Art

[0003] With the widespread adoption and advancement of automotive technology, voice interaction has become the mainstream method for human-computer interaction in vehicle systems. Car owners can now operate the text and buttons on the vehicle's screen by speaking them out loud instead of using their fingers. Related technologies primarily rely on natural language processing (NLP) of the collected interactive speech, supplemented by a large number of rules and templates, to determine interaction commands through template matching. However, this approach suffers from low interaction efficiency and accuracy.

[0004] Summary of the Invention

[0005] Embodiments of the present application provide a multimodal interaction method, device, controller, system, automobile, and storage medium to solve the problems of low efficiency and low accuracy of existing voice interaction.

[0006] A multimodal interaction method, comprising:

[0007] Determine the current interactive speech based on the current interactive voice at the interaction moment;

[0008] Determining, according to the current interaction interface corresponding to the interaction moment, a current scene image and current scene data corresponding to the current interaction interface;

[0009] Using a multimodal recognition model, multimodal recognition is performed on the current interactive speech, the current scene image, and the current scene data to determine a target control instruction;

[0010] Execute the target control instruction to complete the human-computer interaction operation.

[0011] Preferably, the adopting of a multimodal recognition model to perform multimodal recognition on the current interactive speech, the current scene image, and the current scene data to determine the target control instruction includes:

[0012] Determining a multimodal fusion feature according to the current interactive speech, the current scene image, and the current scene data;

[0013] An intention classification model is used to perform intent recognition on the multimodal fusion features to determine target control instructions.

[0014] Preferably, the current scene data includes a current scene text and a current scene code;

[0015] The performing feature fusion on the current interactive speech, the current scene image, and the current scene data to obtain a multimodal fusion feature includes:

[0016] Acquire a first fusion feature according to the current interactive speech, the current scene image, and the current scene text, where the first fusion feature includes position information of a required operation interface element;

[0017] Acquire a second fusion feature according to the current interaction speech and the current scene code, where the second fusion feature includes code information of a required operation interface element;

[0018] The multimodal fusion feature includes the first fusion feature and the second fusion feature.

[0019] Preferably, the current scene text includes the region block names and region block coordinates corresponding to all current interface elements;

[0020] The obtaining of a first fusion feature according to the current interactive speech, the current scene image, and the current scene text includes:

[0021] Matching the current interactive speech with the region block names corresponding to all current interface elements, and determining a first matching degree corresponding to each current interface element;

[0022] If the first matching degree corresponding to the current interface element is greater than a first matching threshold, the current interface element is determined as the required operation interface element, and the coordinates of the region block corresponding to the required operation interface element are obtained;

[0023] Coordinate regression processing is performed on the coordinates of the region block corresponding to the required operation interface element and the current scene image to determine a first fusion feature.

[0024] Preferably, the current scene code includes code function annotations corresponding to all current interface elements and scene codes corresponding to the code function annotations;

[0025] The determining of the second fusion feature according to the current interaction speech and the current scene code includes:

[0026] Matching the current interaction speech with all code function annotations corresponding to the current scene code, and determining a second matching degree corresponding to each code function annotation;

[0027] The scene code for which the second matching degree is greater than a second matching threshold is determined as the second fusion feature.

[0028] Preferably, an intention classification model is used to perform intention recognition on the multimodal fusion features to determine the target control instruction, including:

[0029] Performing intent recognition on the multimodal fusion features using an intent classification model to determine at least one pending control instruction corresponding to the required operation interface element and a recognition score corresponding to the pending control instruction;

[0030] The pending control instruction with the largest recognition score is determined as the target control instruction.

[0031] Preferably, the performing feature fusion on the current interactive speech, the current scene image and the current scene data to obtain multimodal fusion features includes:

[0032] Performing format conversion on the current interactive speech, the current scene image, and the current scene data to obtain standard interactive speech, standard scene image, and standard scene data, respectively;

[0033] Feature fusion is performed on the standard interactive speech, the standard scene image, and the standard scene data to obtain multimodal fusion features.

[0034] A multimodal interaction device, comprising:

[0035] The current interaction speech determination module is used to determine the current interaction speech according to the current interaction voice at the interaction moment;

[0036] A current scene information determination module, configured to determine, based on the current interaction interface corresponding to the interaction moment, a current scene image and current scene data corresponding to the current interaction interface;

[0037] a target control instruction determination module, configured to use a multimodal recognition model to perform multimodal recognition on the current interactive speech, the current scene image, and the current scene data to determine a target control instruction;

[0038] The target control instruction execution module is used to execute the target control instruction to complete the human-computer interaction operation.

[0039] Preferably, the target control instruction determination module includes:

[0040] a multimodal fusion feature determination submodule, configured to determine a multimodal fusion feature based on the current interactive speech, the current scene image, and the current scene data;

[0041] The target control instruction determination submodule is used to use the intention classification model to perform intent recognition on the multimodal fusion features and determine the target control instruction.

[0042] Preferably, the current scene data includes a current scene text and a current scene code;

[0043] The multimodal fusion feature determination submodule includes:

[0044] a first fusion feature determination unit, configured to obtain a first fusion feature according to the current interactive speech, the current scene image, and the current scene text, wherein the first fusion feature includes position information of a required operation interface element;

[0045] A second fusion feature determination unit is configured to obtain a second fusion feature according to the current interaction speech and the current scene code, where the second fusion feature includes code information of a required operation interface element;

[0046] The multimodal fusion feature includes the first fusion feature and the second fusion feature.

[0047] Preferably, the current scene text includes the region block names and region block coordinates corresponding to all current interface elements;

[0048] The first fusion feature determination unit includes:

[0049] A first matching degree determination subunit is configured to match the current interactive speech with the region block names corresponding to all current interface elements, and determine a first matching degree corresponding to each current interface element;

[0050] a region block coordinate determination subunit, configured to determine the current interface element as the required operation interface element if the first matching degree corresponding to the current interface element is greater than a first matching threshold, and obtain the region block coordinates corresponding to the required operation interface element;

[0051] The coordinate regression processing subunit is used to perform coordinate regression processing on the coordinates of the area block corresponding to the required operation interface element and the current scene image to determine the first fusion feature.

[0052] Preferably, the current scene code includes code function annotations corresponding to all current interface elements and scene codes corresponding to the code function annotations;

[0053] The second fusion feature determination unit includes:

[0054] A second matching degree determination subunit is configured to match the current interaction speech with all code function annotations corresponding to the current scene code, and determine a second matching degree corresponding to each code function annotation;

[0055] The second fusion feature determination subunit is configured to determine the scene code for which the second matching degree is greater than a second matching threshold as the second fusion feature.

[0056] Preferably, the target control instruction determination submodule includes:

[0057] a pending control instruction determination unit, configured to perform intent recognition on the multimodal fusion feature using an intent classification model, and determine at least one pending control instruction corresponding to the required operation interface element and a recognition score corresponding to the pending control instruction;

[0058] The target control instruction determining unit is configured to determine the pending control instruction with the largest recognition score as the target control instruction.

[0059] Preferably, the multimodal fusion feature determination submodule includes:

[0060] a format conversion unit, configured to convert the formats of the current interactive speech, the current scene image, and the current scene data to obtain standard interactive speech, standard scene image, and standard scene data, respectively;

[0061] A feature fusion unit is used to perform feature fusion on the standard interactive speech, the standard scene image and the standard scene data to obtain multimodal fusion features.

[0062] A vehicle-mounted controller includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the multimodal interaction method is implemented.

[0063] A vehicle-mounted system comprises the above-mentioned vehicle-mounted controller, a voice acquisition device connected to the vehicle-mounted controller, and a display screen; the voice acquisition device is used to send the collected current interactive voice to the vehicle-mounted controller; and the display screen is used to display the current scene image.

[0064] A car comprises the above-mentioned vehicle-mounted system.

[0065] A computer-readable storage medium stores computer-readable instructions, which implement the multimodal interaction method when executed by a processor.

[0066] The above-mentioned multimodal interaction method, device, controller, system, automobile and storage medium first obtain multimodal data that can reflect the user's human-computer interaction process, such as the current interaction speech, current scene image and current scene data corresponding to the same interaction moment, and input the multimodal data into a pre-trained multimodal recognition model. It can directly output the required target control instructions, which helps to ensure the output efficiency and accuracy of the target control instructions, and can reduce the complex control logic such as customized templates, rules and associations during the development process, thereby improving the adaptability and generalization ability of voice interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0068] FIG1 is a schematic diagram of an application environment of a multimodal interaction method according to an embodiment of the present application;

[0069] FIG2 is a flow chart of a multimodal interaction method according to an embodiment of the present application;

[0070] FIG3 is another flow chart of a multimodal interaction method according to an embodiment of the present application;

[0071] FIG4 is another flow chart of a multimodal interaction method according to an embodiment of the present application;

[0072] FIG5 is another flow chart of a multimodal interaction method according to an embodiment of the present application;

[0073] FIG6 is another flow chart of a multimodal interaction method according to an embodiment of the present application;

[0074] FIG7 is another flow chart of a multimodal interaction method according to an embodiment of the present application;

[0075] FIG8 is a principle block diagram of a multimodal interaction device in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0077] The multimodal interaction method provided in the embodiment of the present application can be applied to a vehicle-mounted system, wherein the vehicle-mounted system includes an on-board controller and a voice acquisition device and a display screen that are communicatively connected to the on-board controller; the on-board controller refers to a controller provided on the vehicle, which can be a controller dedicated to realizing multimodal interaction, or a controller that integrates multimodal interaction functions and other functions. The voice acquisition device is a device for acquiring interactive voice, which can be, but not limited to, a microphone. The display screen is a device for displaying an interactive interface, which can be a touch screen display, which can display a human-computer interaction interface, and can also receive control instructions generated by a user performing touch operations on the human-computer interaction interface, and perform corresponding control operations according to the control instructions.

[0078] In one embodiment, as shown in FIG1 , a multimodal interaction method is provided. The method is described using an onboard controller of a vehicle-mounted system as an example, and includes the following steps:

[0079] S101: Determine the current interaction speech according to the current interaction voice at the interaction moment;

[0080] S102: Determine, based on the current interaction interface corresponding to the interaction moment, a current scene image and current scene data corresponding to the current interaction interface;

[0081] S103: Using a multimodal recognition model, perform multimodal recognition on the current interactive speech, the current scene image, and the current scene data to determine the target control instruction;

[0082] S104: Execute the target control instruction to complete the human-computer interaction operation.

[0083] The current interactive speech refers to the real-time collected speech used to implement human-computer interaction. The current interactive speech is the speech of the user determined by speech recognition of the current interactive speech. The interaction moment refers to the moment when the current interactive speech is collected.

[0084] In one embodiment, in step S101, the vehicle controller may receive the current interactive speech at the interaction moment, which is captured by a voice capture device (including but not limited to a microphone) installed in the vehicle. The current interactive speech can be understood as the speech captured by the user expressing their control intention in a spoken manner. After receiving the current interactive speech, the vehicle controller may perform speech recognition on the current interactive speech using a pre-set speech recognition tool, for example, using an ASR recognition tool to recognize the current interactive speech and obtain the current interactive speech in text form.

[0085] The current interactive interface refers to the interactive interface currently displayed on the vehicle system's display screen. The current scene image refers to the interface image corresponding to the current interactive interface, which displays at least one current interface element. The current interface element is the interface element displayed in the current scene image. The interface element here can be understood as a control used to achieve human-computer interaction, such as an input box, button, scroll bar, and drop-down box displayed on the current interactive interface. The current scene data refers to the tree structure corresponding to the current interactive interface. The tree structure includes nodes corresponding to the current interface elements, and each node includes scene text, scene location, node type, and other data.

[0086] As an example, in step 102, after receiving the current interactive speech, the onboard controller may determine the interactive interface corresponding to the interaction moment as the current interactive interface based on the interaction moment determined by the current interactive speech. That is, the current interactive interface can be understood as the interactive interface when the user expresses their current interactive speech. Then, the current scene image and current scene data corresponding to the current interactive interface are obtained through a pre-set interface. For example, the current scene image corresponding to the current interactive interface can be read through a pre-set API interface, and the current scene data corresponding to the current interactive interface can be read through a pre-set system service interface.

[0087] The multimodal recognition model is used to identify and process multimodal data to determine its control instructions. Multimodal data here refers to data collected through different methods or perspectives for a single object to be described, such as image data and text data representing the same object.

[0088] As an example, the multimodal recognition model is a recognition model determined by pre-training a model using multimodal training samples, and the multimodal training samples include multimodal training data and training control instructions corresponding to the multimodal training data. The multimodal training data includes training interactive speech, training scene images, and training scene data; the training interactive speech here refers to the speech determined by identifying interactive voice during the model training process; the training scene image refers to the scene image corresponding to the interactive interface determined during the model training process; the training scene data refers to the scene data corresponding to the interactive interface determined during the model training process; and the training control instructions refer to the control instructions that match the training interactive speech, training scene image, and training scene data. In this example, the training interactive speech, training scene image, training scene data, and training control instructions are used as multimodal training samples to input a pre-set neural network model for model training, so that the trained multimodal recognition model can output corresponding control instructions based on the input multimodal data (including but not limited to interactive speech, scene images, and scene data).

[0089] For example, the multimodal training samples are as follows: query: training interactive speech; sceneData: training scene data; Image: training scene image; Target: training control instruction; query: multimedia brightness is adjusted to 60%. The specific examples are as follows:

[0090] As an example, in step S103, the on-board controller can input multimodal data such as the current interactive speech, the current scene image, and the current scene data into a pre-trained multimodal recognition model, and can quickly obtain the target control instructions output by it. Compared with the traditional method of simply using the current interactive speech to locate the scene information of the current interactive interface, the current interactive speech is combined with the current scene image and the current scene data to quickly and accurately locate the focus interface element that the user needs to manipulate, so as to output the target control instruction corresponding to the focus interface element, which helps to ensure the output efficiency and accuracy of the target control instruction; compared with the traditional method of requiring a large number of rules and templates for conversion control process, the pre-trained multimodal recognition model is used to identify multimodal data such as the current interactive speech, the current scene image, and the current scene data, and can directly output the required target control instruction, reducing the complex control logic such as customized templates, rules and associations during the development process, and helping to improve the adaptability and generalization ability of voice interaction.

[0091] In this example, the target control instruction determined includes: control type, control location, control ID, text, and action. The actions here include but are not limited to click, long press, set value, set slide, and set text. For example, in the above target control instruction "rect:[308, 414, 338, 1073]|id:211|action:setValue|value:60%", the control location is "[308, 414, 338, 1073]|", the control ID is "id:211", the text is "action:setValue", and the action is "set value value:60%".

[0092] As an example, in step S104, the vehicle controller can complete the human-computer interaction operation according to the target control instructions directly output by the multimodal recognition model.

[0093] In this embodiment, multimodal data that can reflect the user's human-computer interaction process, such as the current interaction words, current scene image and current scene data corresponding to the same interaction moment, are first obtained, and the multimodal data is input into a pre-trained multimodal recognition model. The required target control instructions can be directly output, which helps to ensure the output efficiency and accuracy of the target control instructions, and can reduce the complex control logic such as customized templates, rules and associations in the development process, thereby improving the adaptability and generalization ability of voice interaction.

[0094] In one embodiment, as shown in FIG2 , step S103 , i.e., using a multimodal recognition model to perform multimodal recognition on the current interactive speech, the current scene image, and the current scene data to determine the target control instruction, includes:

[0095] S201: Determine multimodal fusion features based on current interactive speech, current scene image, and current scene data;

[0096] S202: Use the intent classification model to identify the multimodal fusion features and determine the target control instructions.

[0097] As an example, a multimodal recognition model includes a feature fusion model and an intent classification model. The feature fusion model is used to implement feature extraction and fusion processing. It can extract and fuse features from input multimodal data to output fused features corresponding to the multimodal data. The intent classification model is used to identify the intent of the features corresponding to the multimodal data and output corresponding control instructions.

[0098] As an example, in step S201, after acquiring multimodal data such as the current interaction speech, the current scene image, and the current scene data, the onboard controller can use a pre-trained feature fusion model to extract and fuse features from the current interaction speech, the current scene image, and the current scene data, thereby rapidly extracting multimodal fusion features that reflect the user's intent. In this example, compared to the traditional method of simply using the current interaction speech to globally match all scene data in the vehicle system, the current interaction speech is combined with the current scene image and the current scene data for local matching, which can quickly and accurately extract multimodal fusion features corresponding to the current interaction speech.

[0099] As an example, in step S202, after obtaining multimodal fusion features extracted from multimodal data such as the current interactive speech, the current scene image, and the current scene data, the on-board controller can input the multimodal fusion features into a pre-trained intent classification model for identification, and obtain the target control instructions output by the intent classification model. In this example, the intent classification model performs intent recognition on the multimodal fusion features, determines the target intent corresponding to the current interactive speech, and directly outputs the target control instructions corresponding to the target intent. Compared to the traditional control method of first outputting the target intent and then using a large number of rules and templates for conversion, this can effectively reduce the complex control logic such as customized templates, rules, and associations during the development process, helping to improve the adaptability and generalization capabilities of voice interaction.

[0100] In this embodiment, feature extraction and fusion are first performed on multimodal data such as the current interactive speech, current scene image and current scene data, so as to quickly and accurately obtain multimodal fusion features that represent user intentions; then, an intention classification model is used to identify the intentions of the multimodal fusion features, and target control instructions are directly output, omitting the process of first outputting the target intention and then using a large number of rules and templates for conversion. This can effectively reduce the complex control logic such as customized templates, rules and associations during the development process, and help improve the adaptability and generalization ability of voice interaction.

[0101] In one embodiment, the current scene data includes a current scene text and a current scene code;

[0102] As shown in FIG3 , step S201 is to perform feature fusion on the current interactive speech, the current scene image, and the current scene data to obtain multimodal fusion features, including:

[0103] S301: Obtain a first fusion feature based on the current interactive speech, the current scene image, and the current scene text, where the first fusion feature includes position information of a required operation interface element;

[0104] S302: Obtain a second fusion feature based on the current interaction speech and the current scene code, where the second fusion feature includes code information of a required operation interface element;

[0105] The multimodal fusion feature includes a first fusion feature and a second fusion feature.

[0106] Among them, the current scene text is used to record the area block information corresponding to the current interface element displayed in the current scene image. The current interface element here is the interface element displayed in the current scene image, which is an interface element that can perform human-computer interaction with the user. Each area block information includes area block coordinates and area block name. The area block name is the name of the area block where a current interface element is located in the current scene data. The area block coordinates are the coordinates of the area block where a current interface element is located in the current scene data. For example, when the current scene image contains current interface elements corresponding to multiple menu areas such as "Audio", "Display", "System", "Basic Management" and "Application Management", the current scene text extracted through the system service interface includes the area block information corresponding to the above menu areas, specifically including area block coordinates and area block names.

[0107] The current scene code is used to record the code corresponding to the current interface element displayed in the current scene image. For example, in order to implement multimedia brightness adjustment, in addition to displaying the current interface element of brightness adjustment on the current scene image for user touch operation, its current scene data will record the current scene code that implements this function. For example, the current scene code is: position:Multimedia brightness adjustment|rect[624,278,414,237]-seekBar|rect[308,414,338,1073];type:TextView-SeekBar;slide:false-true;id:209-211".

[0108] The required operation interface element refers to the current interface element that needs to be operated as indicated by the current interactive dialogue.

[0109] As an example, in step S301, after acquiring multimodal data, the onboard controller may perform feature matching on the current interactive speech, the current scene image, and the current scene text in the current scene data to determine a first fused feature corresponding to the location information of the interface element required for the current interactive speech. This first fused feature is a feature fused from the three modal data, specifically a feature formed by including the block information corresponding to the interface element required for the current interactive speech, thereby achieving the purpose of quickly focusing on and locating the required interface element. For example, the onboard controller may perform matching processing on the current scene text corresponding to all current interface elements displayed in the current interactive speech and the current scene image, and determine the block information of the successfully matched current interface elements in the current scene image, thereby achieving the purpose of quickly focusing on the required interface element. The required interface element here refers to the current interface element determined to be required for operation by the current interactive speech, for example, a current interface element selected from the multiple current interface elements such as "audio", "display", "system", "basic management", and "application management" mentioned above.

[0110] As an example, in step S302, after acquiring multimodal data, the on-board controller may perform feature matching on the current interactive speech and the current scene code in the current scene data to determine the second fusion feature corresponding to the code information of the operation interface element required for the current interactive speech. The second fusion feature is the feature after the fusion of the above two modal data, specifically the feature formed by the scene code corresponding to the operation interface element required for the current interactive speech, so as to achieve targeted determination of the scene code of the current interface element, and provide a guarantee for subsequent intention recognition and direct output of the corresponding target control instruction. For example, the current scene code records the scene code of all current interface elements displayed in the current scene image, and the current interactive speech can be matched with the scene code of all current interface elements, and the scene code corresponding to the required operation interface element can be quickly determined, thereby obtaining the second fusion feature corresponding to the code information of the required operation interface element.

[0111] In this embodiment, based on the current interactive speech, the current scene image and the current scene text, the first fusion feature corresponding to the position information of the operating interface element required for the current interactive speech can be quickly determined. Compared with the traditional method of simply using the current interactive speech to globally match all scene data of the vehicle system, the scene data that needs to be matched determined based on the current scene image and the current scene text is less, and the first fusion feature corresponding to the position information of the required operating interface element of the required operating interface element can be quickly and accurately determined; based on the current interactive speech and the current scene code, the second fusion feature corresponding to the code information of the operating interface element required for the current interactive speech can be quickly determined. Compared with the traditional method of simply using the current interactive speech to globally match all scene codes of the vehicle system layer by layer, since the scene codes that need to be matched determined by the current scene image are less, the second fusion feature corresponding to the code information of the required operating interface element can be quickly and accurately determined.

[0112] In one embodiment, the current scene text includes the region block names and region block coordinates corresponding to all current interface elements;

[0113] As shown in FIG4 , step S301, i.e., obtaining a first fusion feature based on the current interactive speech, the current scene image, and the current scene text, includes:

[0114] S401: Match the current interactive speech with the region block names corresponding to all current interface elements, and determine a first matching degree corresponding to each current interface element;

[0115] S402: If the first matching degree corresponding to the current interface element is greater than the first matching threshold, the current interface element is determined as the required operation interface element, and the coordinates of the region block corresponding to the required operation interface element are obtained;

[0116] S403: Performing coordinate regression processing on the coordinates of the region block corresponding to the required operation interface element and the current scene image to determine a first fusion feature.

[0117] The first matching degree is the matching degree between the current interactive speech and the region block name. The first matching threshold is a pre-set first threshold for evaluating whether a relatively matching standard is met.

[0118] As an example, in step S401, the onboard controller may match the current interactive speech with the region block names corresponding to all current interface elements in the current scene text to determine a first matching degree corresponding to each current interface element. For example, the onboard controller may use, but is not limited to, a text matching algorithm such as Jaccard similarity, Levenshtein edit distance, Simhash, TF-IDF, and BM25 to match the current interactive speech with the region block names corresponding to all current interface elements to determine a first matching degree corresponding to each current interface element.

[0119] As an example, in step S402, after determining the first matching degree corresponding to each current interface element, the on-board controller can compare the first matching degree with a pre-set first matching threshold. If the first matching degree is greater than the first matching threshold, it is determined that the area block name of the current interface element has a high degree of matching with the current interactive language, and the current interface element corresponding to the area block name can be determined as the required operation interface element, and then the area block coordinates corresponding to the required operation interface element are determined.

[0120] As an example, in step S403, after determining the area block coordinates corresponding to the required operation interface element, the on-board controller can use a preset coordinate regression algorithm to perform coordinate regression processing on the area block coordinates corresponding to the required operation interface element and the current scene image to obtain a first fusion feature containing the position information of the required operation interface element.

[0121] The onboard controller may use target detection algorithms such as, but not limited to, Faster-RCNN, RetinaNet, and SSD to perform target detection on the current scene image and the coordinates of the area blocks corresponding to the desired operating interface elements to determine a set of prediction frames. For example, after determining the coordinates of the area blocks corresponding to the desired operating interface elements, a basic prediction frame anchor may be generated based on the area block coordinates; the coordinates of the basic prediction frame anchor may be transformed into the form of a center point, width w, and height h; the width w and height h may be used to calculate the rectangular area corresponding to the basic prediction frame anchor; based on the rectangular area and the preset aspect ratio parameter rations, three first prediction frame anchors may be obtained; and all first prediction frame anchors may be expanded using the preset area ratio parameter scale to obtain a prediction frame set formed by nine second prediction frame anchors. The onboard controller may then use, but not limited to, the kmeans clustering algorithm to perform coordinate regression processing on the prediction frame set to obtain a first fused feature containing the location information of the desired operating interface element. For example, you can first extract all the bboxes corresponding to the current scene image; select the width w and height h of all bboxes; randomly select k values ​​of w and h as initial values ​​from the 9 second prediction box anchors; calculate the 9 iou values ​​corresponding to each bbox and the 9 second prediction box anchors; determine the second prediction box anchor corresponding to the minimum value of the 9 iou values ​​as the target anchor corresponding to the bbox, and assign the bbox to the cluster where its corresponding target anchor is located. Repeat the above steps until the classification cluster does not change after the anchors are updated, and complete the coordinate regression operation, so that the final second fusion feature can accurately locate the target prediction box corresponding to the required operation interface element in the current scene image, and determine the position information corresponding to the target prediction box as the first fusion feature.

[0122] In this embodiment, the current interactive speech is first matched with the area block names of all current interface elements corresponding to the current scene image to determine the required operation interface elements with a higher degree of matching. Since the number of current interface elements displayed in the current scene image is limited, the matching process is faster, which helps to improve the efficiency of obtaining the required operation interface elements; the coordinates of the area block corresponding to the required operation interface elements and the current scene image are used for coordinate regression processing, which can ensure the efficiency of obtaining the first fusion feature containing the position information of the required operation interface elements, thereby improving the overall processing efficiency and positioning accuracy.

[0123] In one embodiment, the current scene code includes code function annotations corresponding to all current interface elements and scene codes corresponding to the code function annotations;

[0124] As shown in FIG5 , step S302, i.e., determining a second fusion feature based on the current interaction speech and the current scene code, includes:

[0125] S501: Match the current interaction speech with all code function annotations corresponding to the current scene code, and determine a second matching degree corresponding to each code function annotation;

[0126] S502: Determine the scene code with a second matching degree greater than a second matching threshold as a second fusion feature.

[0127] Code function annotations refer to textual content that uses natural language to describe the functionality of code content. Scenario code refers to the code content used to implement a specific scenario function. The first match degree is the degree of match between the current interaction language and the code function annotations. The second match threshold is a pre-set threshold used to assess whether a higher match standard has been met.

[0128] As an example, in step S501, the onboard controller may match the current interaction speech with all code function annotations in the current scene code to determine a first matching degree corresponding to each code function annotation. For example, the onboard controller may use, but is not limited to, a text matching algorithm such as Jaccard similarity, Levenshtein edit distance, Simhash, TF-IDF, and BM25 to match the current interaction speech with all code function annotations to determine a second matching degree corresponding to each code function annotation.

[0129] As an example, in step S502, after determining the second matching degree corresponding to each code function annotation, the on-board controller can compare the second matching degree with a pre-set second matching threshold. If the second matching degree is greater than the second matching threshold, it is determined that the code function annotation has a high degree of matching with the current interactive dialogue. The scene code corresponding to the code function annotation can be determined as the code information corresponding to the required operation interface element to obtain the second fusion feature.

[0130] For example, the current scenario code includes Code Block 1 and Code Block 2. Code Block 1 is "position: Multimedia brightness adjustment|rect[624,278,414,237]-seekBar|rect[308,414,338,1073]; type: TextView - SeekBar; slide: false - true; id: 209 - 211", and Code Block 2 is "position: Multimedia screensaver|rect[428,414,469,564]-Clock 1|rect[496,580,550,743]; type: TextView - RadioButton; click: false - true; id: 215 - 216". When the current interaction statement is "Multimedia brightness adjustment is 60%", the current interaction statement can be matched with code function annotations such as "Multimedia brightness adjustment" and "Multimedia screensaver" to determine the second matching degree A corresponding to "Multimedia brightness adjustment" and the second matching degree B corresponding to "Multimedia screensaver"; compare the second matching degree A and the second matching degree B with the second matching threshold S respectively. If A > S and B < S, then Code Block 1 corresponding to the second matching degree A can be determined as the code information corresponding to the required operation interface element and determined as the second fusion feature.

[0131] In this embodiment, first, the current interaction statement is matched with the code function annotations in the current scenario code to determine the scenario code corresponding to the code function annotation with a higher matching degree, and it is determined as the code information corresponding to the required operation interface element to obtain the second fusion feature. Since the number of current interface elements displayed in the current scenario image is limited, the matching process is relatively fast, which helps to improve the acquisition efficiency of determining the required operation interface element, and further ensures the acquisition efficiency of the second fusion feature of the code information of the required operation interface element, thereby improving the overall processing efficiency.

[0132] In one embodiment, as shown in FIG. 6, step S201, that is, performing feature fusion on the current interaction statement, the current scenario image, and the current scenario data to obtain a multi-modal fusion feature, includes:

[0133] S601: Performing format conversion on the current interaction statement, the current scenario image, and the current scenario data to respectively obtain a standard interaction statement, a standard scenario image, and standard scenario data;

[0134] [[ID=​​The standard interactive speech is the speech determined after the current interactive speech is converted. The standard scene image is the scene image determined after the current scene image is converted, for example, it can be a scene image in RGB format. The standard scene data is the scene data determined after the current scene data is converted.

[0136] As an example, in S601, after obtaining multimodal data, the on-board controller can use a preset format conversion logic to convert the current interactive speech, current scene image and current scene data respectively to obtain standard interactive speech, standard scene image and standard scene data in a standard format, so that the multimodal data after format conversion is the same as the data format used in the multimodal recognition model training process.

[0137] As an example, in step S602, after acquiring multimodal data in a standard format, including standard interaction dialogue, standard scene images, and standard scene data, the onboard controller can input this standard multimodal data into a pre-trained multimodal recognition model for multimodal recognition to determine the target control instructions. In this example, using the multimodal recognition model to recognize the standard multimodal data helps ensure the efficiency and accuracy of the output target control instructions.

[0138] In one embodiment, as shown in FIG7 , step S202 , using an intent classification model to perform intent recognition on multimodal fusion features to determine a target control instruction, includes:

[0139] S701: Using an intent classification model to perform intent recognition on the multimodal fusion features, and determining at least one pending control instruction corresponding to a required operation interface element and a recognition score corresponding to the pending control instruction;

[0140] S702: Determine the pending control instruction with the largest recognition score as the target control instruction.

[0141] The pending control instruction refers to the control instruction determined by intent classification of multimodal fusion features using the intent classification model. The target control instruction is the control instruction with the highest recognition score among all pending control instructions.

[0142] As an example, in step S701, the on-board controller uses an intent classification model to perform intent recognition on the multimodal fusion features formed by fusing the current interactive speech, the current scene image and the current scene data. It can determine at least one pending control instruction corresponding to the required operation interface element displayed in the current scene image, and determine the recognition score corresponding to each pending control instruction. The recognition score can be understood as the probability that the pending control instruction is the actual control instruction corresponding to the current interactive speech.

[0143] As an example, in step S702 , after determining the identification score corresponding to at least one pending control instruction, the onboard controller may compare the at least one identification score to determine the pending control instruction with the largest identification score as the target control instruction.

[0144] In this embodiment, the intention classification model is first used to identify the multimodal fusion features, determine the pending control instructions corresponding to each interface element in the current interactive interface and their identification scores, and determine the pending control instruction with the largest identification score as the target control instruction corresponding to the current interactive speech. Since the number of current interface elements in the current scene image is limited, the pending control instructions identified are more targeted, which helps to ensure the efficiency and accuracy of the target control instructions finally output.

[0145] Compared with the existing technology that uses interactive language to globally match the text content of all scene data in the vehicle system, and then determines the control instructions based on the text content, which has the problems of low efficiency and low accuracy, this solution integrates interactive language, scene images and scene data to obtain an effect similar to the coordination of mouth and eyes, with the following beneficial effects: First, this solution can better understand user intentions and directly hit the required operating interface elements, which helps to ensure its processing efficiency and accuracy; second, the multimodal recognition model in this solution will directly output the target control instructions corresponding to the target intention. Compared with the traditional method of using a large number of rules and templates to convert the target intention to determine the final target control instructions, its processing process is more precise to improve its processing effect, and can reduce complex control logic such as customized templates, rules and associations in the development process, thereby improving the adaptability and generalization ability of voice interaction. Third, since the multimodal recognition model is a model formed by pre-model training, it has a certain generalization ability. It can be used for the current scene image and current scene data with corresponding relationships. Compared with the existing technology, due to system changes, third-party incoordination or system component changes, it will cause the interface to change and require re-customization of rules and templates. It has better versatility and can still be used directly after the car system is upgraded. It can greatly save the cost of product development and operation, and is not affected by the system model. In theory, it can be used for Linux, Android, and IOS.

[0146] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0147] In one embodiment, a multimodal interaction device is provided, which corresponds one-to-one to the multimodal interaction method in the above embodiment. As shown in Figure 8, the multimodal interaction device includes a current interaction speech determination module 801, a current scene information determination module 802, a target control instruction determination module 803, and a target control instruction execution module 804. The functional modules are described in detail as follows:

[0148] The current interaction speech determination module 801 is used to determine the current interaction speech according to the current interaction voice at the interaction moment;

[0149] The current scene information determination module 802 is configured to determine, based on the current interaction interface corresponding to the interaction moment, the current scene image and current scene data corresponding to the current interaction interface;

[0150] The target control instruction determination module 803 is configured to use a multimodal recognition model to perform multimodal recognition on the current interactive speech, the current scene image, and the current scene data to determine a target control instruction;

[0151] The target control instruction execution module 804 is used to execute the target control instruction to complete the human-computer interaction operation.

[0152] In one embodiment, the target control instruction determination module 803 includes:

[0153] a multimodal fusion feature determination submodule, configured to determine a multimodal fusion feature based on the current interactive speech, the current scene image, and the current scene data;

[0154] The target control instruction determination submodule is used to use the intention classification model to perform intent recognition on the multimodal fusion features and determine the target control instruction.

[0155] In one embodiment, the current scene data includes a current scene text and a current scene code;

[0156] The multimodal fusion feature determination submodule includes:

[0157] a first fusion feature determination unit, configured to obtain a first fusion feature according to the current interactive speech, the current scene image, and the current scene text, wherein the first fusion feature includes position information of a required operation interface element;

[0158] A second fusion feature determination unit is configured to obtain a second fusion feature according to the current interaction speech and the current scene code, where the second fusion feature includes code information of a required operation interface element;

[0159] The multimodal fusion feature includes the first fusion feature and the second fusion feature.

[0160] In one embodiment, the current scene text includes the region block names and region block coordinates corresponding to all current interface elements;

[0161] The first fusion feature determination unit includes:

[0162] A first matching degree determination subunit is configured to match the current interactive speech with the region block names corresponding to all current interface elements, and determine a first matching degree corresponding to each current interface element;

[0163] a region block coordinate determination subunit, configured to determine the current interface element as the required operation interface element if the first matching degree corresponding to the current interface element is greater than a first matching threshold, and obtain the region block coordinates corresponding to the required operation interface element;

[0164] The coordinate regression processing subunit is used to perform coordinate regression processing on the coordinates of the area block corresponding to the required operation interface element and the current scene image to determine the first fusion feature.

[0165] In one embodiment, the current scene code includes code function annotations corresponding to all current interface elements and scene codes corresponding to the code function annotations;

[0166] The second fusion feature determination unit includes:

[0167] A second matching degree determination subunit is configured to match the current interaction speech with all code function annotations corresponding to the current scene code, and determine a second matching degree corresponding to each code function annotation;

[0168] The second fusion feature determination subunit is configured to determine the scene code for which the second matching degree is greater than a second matching threshold as the second fusion feature.

[0169] In one embodiment, the target control instruction determination submodule includes:

[0170] a pending control instruction determination unit, configured to perform intent recognition on the multimodal fusion feature using an intent classification model, and determine at least one pending control instruction corresponding to the required operation interface element and a recognition score corresponding to the pending control instruction;

[0171] The target control instruction determining unit is configured to determine the pending control instruction with the largest recognition score as the target control instruction.

[0172] In one embodiment, the multimodal fusion feature determination submodule includes:

[0173] a format conversion unit, configured to convert the formats of the current interactive speech, the current scene image, and the current scene data to obtain standard interactive speech, standard scene image, and standard scene data, respectively;

[0174] A feature fusion unit is used to perform feature fusion on the standard interactive speech, the standard scene image and the standard scene data to obtain multimodal fusion features.

[0175] For the specific definition of the multimodal interaction device, please refer to the definition of the multimodal interaction method above and will not be repeated here. The various modules in the above-mentioned multimodal interaction device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0176] In one embodiment, a vehicle-mounted controller is provided, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the multimodal interaction method in the above-mentioned embodiment is implemented, such as S101-S103 shown in FIG1 , or as shown in FIG2 to FIG7 . To avoid repetition, they are not described here. Alternatively, when the computer-readable instructions are executed by the processor, the functions of the modules / units in the above-mentioned multimodal interaction device embodiment are implemented, such as the functions of the current interaction speech determination module 801, the current scene information determination module 802, the target control instruction determination module 803, and the target control instruction execution module 804 shown in FIG8 . To avoid repetition, they are not described here.

[0177] A vehicle-mounted system includes the above-mentioned on-board controller, a voice acquisition device connected to the on-board controller, and a display screen; the voice acquisition device is used to send the collected current interactive voice to the on-board controller; and the display screen is used to display the current scene image. In this example, the voice acquisition device acquires the current interactive voice in real time and sends the current interactive voice to the on-board controller. After receiving the current interactive voice, the on-board controller can obtain the current scene image and current scene data currently displayed on the display screen through a preset interface, use a multimodal recognition model to identify the current interactive speech, current scene image, and current scene data corresponding to the current interactive voice, and directly output target control instructions so that human-computer interaction operations can be performed based on the target control instructions. This helps to ensure the output efficiency and accuracy of the target control instructions, and can reduce complex control logic such as customized templates, rules, and associations during the development process, thereby improving the adaptability and generalization of voice interaction.

[0178] A car comprises the above-mentioned vehicle-mounted system.

[0179] A computer-readable storage medium stores computer-readable instructions, which implement the multimodal interaction method when executed by a processor.

[0180] In one embodiment, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the multimodal interaction method in the above-mentioned embodiment is implemented, such as S101-S103 shown in FIG1 , or as shown in FIG2 to FIG7 . To avoid repetition, they are not described here. Alternatively, when the computer-readable instructions are executed by a processor, the functions of the modules / units in the above-mentioned multimodal interaction device embodiment are implemented, such as the functions of the current interaction speech determination module 801, the current scene information determination module 802, the target control instruction determination module 803, and the target control instruction execution module 804 shown in FIG8 . To avoid repetition, they are not described here.

[0181] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0182] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0183] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A multimodal interaction method, wherein: include: Determine the current interactive speech based on the current interactive voice at the interaction time; Determining, according to the current interaction interface corresponding to the interaction moment, a current scene image and current scene data corresponding to the current interaction interface; Using a multimodal recognition model, multimodal recognition is performed on the current interactive speech, the current scene image and the current scene data to determine a target control instruction; Execute the target control instruction to complete the human-computer interaction operation.

2. The multimodal interaction method according to claim 1, wherein: The adopting of the multimodal recognition model to perform multimodal recognition on the current interactive speech, the current scene image and the current scene data to determine the target control instruction includes: Determining a multimodal fusion feature according to the current interactive speech, the current scene image and the current scene data; An intention classification model is used to perform intent recognition on the multimodal fusion features to determine the target control instructions.

3. The multimodal interaction method according to claim 2, wherein: The current scene data includes a current scene text and a current scene code; The performing feature fusion on the current interactive speech, the current scene image and the current scene data to obtain multimodal fusion features includes: Acquire a first fusion feature according to the current interactive speech, the current scene image, and the current scene text, where the first fusion feature includes position information of a required operation interface element; Acquire a second fusion feature according to the current interaction speech and the current scene code, where the second fusion feature includes code information of a required operation interface element; The multimodal fusion feature includes the first fusion feature and the second fusion feature.

4. The multimodal interaction method according to claim 3, wherein: The current scene text includes the region block names and region block coordinates corresponding to all current interface elements; The acquiring a first fusion feature according to the current interactive speech, the current scene image and the current scene text includes: Matching the current interactive speech with the area block names corresponding to all current interface elements, and determining a first matching degree corresponding to each current interface element; If the first matching degree corresponding to the current interface element is greater than a first matching threshold, the current interface element is determined as the required operation interface element, and the region block coordinates corresponding to the required operation interface element are obtained; Coordinate regression processing is performed on the region block coordinates corresponding to the required operation interface element and the current scene image to determine a first fusion feature.

5. The multimodal interaction method according to claim 3, wherein: The current scene code includes code function annotations corresponding to all current interface elements and scene codes corresponding to the code function annotations; The determining, according to the current interactive speech and the current scene code, a second fusion feature includes: Matching the current interactive speech with all code function annotations corresponding to the current scene code, and determining a second matching degree corresponding to each of the code function annotations; The scene code for which the second matching degree is greater than a second matching threshold is determined as the second fusion feature.

6. The multimodal interaction method according to claim 3, wherein: The adopting the intention classification model to perform intention recognition on the multimodal fusion feature to determine the target control instruction includes: Using an intention classification model to perform intention recognition on the multimodal fusion feature, and determining at least one pending control instruction corresponding to the required operation interface element and a recognition score corresponding to the pending control instruction; The pending control instruction with the largest recognition score is determined as the target control instruction.

7. The multimodal interaction method according to claim 2, wherein: The performing feature fusion on the current interactive speech, the current scene image and the current scene data to obtain multimodal fusion features includes: Performing format conversion on the current interactive speech, the current scene image and the current scene data to obtain standard interactive speech, standard scene image and standard scene data respectively; Feature fusion is performed on the standard interactive speech, the standard scene image and the standard scene data to obtain multimodal fusion features.

8. A multimodal interaction device, wherein: include: A current interaction speech determination module is used to determine the current interaction speech according to the current interaction voice at the interaction moment; A current scene information determination module, used to determine a current scene image and current scene data corresponding to the current interaction interface according to the current interaction interface corresponding to the interaction moment; A target control instruction determination module is used to use a multimodal recognition model to perform multimodal recognition on the current interactive speech, the current scene image and the current scene data to determine the target control instruction; The target control instruction execution module is used to execute the target control instruction to complete the human-computer interaction operation.

9. The multimodal interaction device according to claim 8, wherein: The target control instruction determination module includes: A multimodal fusion feature determination submodule, used to determine a multimodal fusion feature according to the current interactive speech, the current scene image and the current scene data; The target control instruction determination submodule is used to use the intention classification model to perform intent recognition on the multimodal fusion features and determine the target control instruction.

10. The multimodal interaction device according to claim 9, wherein: The current scene data includes a current scene text and a current scene code; The multimodal fusion feature determination submodule includes: A first fusion feature determination unit is used to obtain a first fusion feature according to the current interactive speech, the current scene image and the current scene text, where the first fusion feature includes position information of a required operation interface element; A second fusion feature determination unit is used to obtain a second fusion feature according to the current interaction speech and the current scene code, where the second fusion feature includes code information of a required operation interface element; The multimodal fusion feature includes the first fusion feature and the second fusion feature.

11. The multimodal interaction device according to claim 10, wherein: The current scene text includes the region block names and region block coordinates corresponding to all current interface elements; The first fusion feature determination unit includes: A first matching degree determination subunit is used to match the current interactive speech with the area block names corresponding to all current interface elements to determine a first matching degree corresponding to each current interface element; A region block coordinate determination subunit, configured to determine the current interface element as a required operation interface element if the first matching degree corresponding to the current interface element is greater than a first matching threshold, and obtain the region block coordinates corresponding to the required operation interface element; The coordinate regression processing subunit is used to perform coordinate regression processing on the coordinates of the area block corresponding to the required operation interface element and the current scene image to determine the first fusion feature.

12. The multimodal interaction device according to claim 9, wherein: The current scene code includes code function annotations corresponding to all current interface elements and scene codes corresponding to the code function annotations; The second fusion feature determination unit includes: A second matching degree determination subunit is used to match the current interactive speech with all code function annotations corresponding to the current scene code, and determine a second matching degree corresponding to each code function annotation; The second fusion feature determination subunit is used to determine the scene code whose second matching degree is greater than a second matching threshold as the second fusion feature.

13. The multimodal interaction device according to claim 10, wherein: The target control instruction determination submodule includes: A pending control instruction determination unit, used to use an intention classification model to perform intent recognition on the multimodal fusion feature, and determine at least one pending control instruction corresponding to the required operation interface element and a recognition score corresponding to the pending control instruction; The target control instruction determination unit is used to determine the pending control instruction with the largest recognition score as the target control instruction.

14. The multimodal interaction device according to claim 9, wherein: The multimodal fusion feature determination submodule includes: A format conversion unit, used to convert the formats of the current interactive speech, the current scene image and the current scene data, and obtain standard interactive speech, standard scene image and standard scene data respectively; A feature fusion unit is used to perform feature fusion on the standard interactive speech, the standard scene image and the standard scene data to obtain multimodal fusion features.

15. A vehicle controller, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein: When the processor executes the computer-readable instructions, the multimodal interaction method according to any one of claims 1 to 7 is implemented.

16. A vehicle system, wherein: It comprises the vehicle-mounted controller as claimed in claim 15, a voice acquisition device and a display screen connected to the vehicle-mounted controller; the voice acquisition device is used to send the collected current interactive voice to the vehicle-mounted controller; the display screen is used to display the current scene image.

17. A car, wherein: Including the vehicle system as described in claim 16.

18. A computer-readable storage medium storing computer-readable instructions, wherein: When the computer-readable instructions are executed by a processor, the multimodal interaction method according to any one of claims 1 to 7 is implemented.

Images