Multimodal interaction method, apparatus, controller, system, automobile, and storage medium

Abstract

The application discloses a multimodal interaction method, device, controller, system, automobile and storage medium. The method comprises the following steps: determining a current interaction dialogue according to a current interaction voice at an interaction time; determining a current scene image and current scene data corresponding to a current interaction interface corresponding to the interaction time; adopting a multimodal recognition model to perform multimodal recognition on the current interaction dialogue, the current scene image and the current scene data, and determining a target control instruction; and executing the target control instruction to complete a human-computer interaction operation. The method can guarantee the output efficiency and accuracy of the target control instruction, reduce the complexity of customizing templates, rules and associations and other complex control logics in the development process, and improve the adaptability and generalization ability of voice interaction.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a multimodal interaction method, device, controller, system, automobile, and storage medium. Background Technology

[0002] With the popularization and development of automotive technology, voice interaction has become the mainstream method for human-computer interaction in in-vehicle systems. Car owners can operate the system by speaking the text and buttons on the screen instead of using their fingers. Related technologies mainly involve Natural Language Processing (NLP) on the collected voice input, supplemented by numerous rules and templates. Interaction commands are determined through template matching. However, this approach suffers from low efficiency and accuracy. Summary of the Invention

[0003] This invention provides a multimodal interaction method, device, controller, system, vehicle, and storage medium to solve the problems of low efficiency and low accuracy in existing voice interaction methods.

[0004] A multimodal interaction method, comprising:

[0005] Determine the current interaction script based on the current interaction voice at the moment of interaction;

[0006] Based on the current interactive interface corresponding to the interaction time, determine the current scene image and current scene data corresponding to the current interactive interface;

[0007] A multimodal recognition model is used to perform multimodal recognition on the current interactive dialogue, the current scene image, and the current scene data to determine the target control command;

[0008] Execute the target control command to complete the human-computer interaction operation.

[0009] Preferably, the step of employing a multimodal recognition model to perform multimodal recognition on the current interactive dialogue, the current scene image, and the current scene data to determine the target control command includes:

[0010] Based on the current interactive dialogue, the current scene image, and the current scene data, determine the multimodal fusion features;

[0011] An intent classification model is used to identify the intent of the multimodal fusion features and determine the target control command.

[0012] Preferably, the current scene data includes current scene text and current scene code;

[0013] The step of fusing features from the current interactive dialogue, the current scene image, and the current scene data to obtain multimodal fusion features includes:

[0014] Based on the current interactive dialogue, the current scene image, and the current scene text, a first fusion feature is obtained, wherein the first fusion feature includes the location information of the required operation interface elements;

[0015] Based on the current interactive dialogue and the current scene code, a second fusion feature is obtained, the second fusion feature containing the code information of the required operation interface elements;

[0016] The multimodal fusion feature includes the first fusion feature and the second fusion feature.

[0017] Preferably, the current scene text includes the names and coordinates of all area blocks corresponding to the current interface elements;

[0018] The step of obtaining the first fusion feature based on the current interactive dialogue, the current scene image, and the current scene text includes:

[0019] The current interactive dialogue is matched with the area block names corresponding to all current interface elements to determine the first matching degree for each current interface element.

[0020] If the first matching degree corresponding to the current interface element is greater than the first matching threshold, then the current interface element is determined as the required operation interface element, and the coordinates of the area block corresponding to the required operation interface element are obtained.

[0021] Coordinate regression processing is performed on the region block coordinates corresponding to the required operation interface elements and the current scene image to determine the first fusion feature.

[0022] Preferably, the current scene code includes code function comments corresponding to all current interface elements and scene code corresponding to the code function comments;

[0023] The step of determining the second fusion feature based on the current interaction dialogue and the current scene code includes:

[0024] The current interactive dialogue is matched with all code function comments corresponding to the current scene code to determine the second matching degree corresponding to each code function comment;

[0025] The scene codes with a second matching degree greater than the second matching threshold are identified as the second fusion feature.

[0026] Preferably, an intent classification model is used to identify the intent of the multimodal fusion features and determine the target control command, including:

[0027] An intent classification model is used to identify the intent of the multimodal fusion features, and at least one pending control instruction corresponding to the required operation interface element and the identification score corresponding to the pending control instruction are determined.

[0028] The control command with the highest identification score is identified as the target control command.

[0029] Preferably, the step of fusing features from the current interactive dialogue, the current scene image, and the current scene data to obtain multimodal fusion features includes:

[0030] The current interactive dialogue, the current scene image, and the current scene data are converted into formats to obtain standard interactive dialogue, standard scene image, and standard scene data, respectively.

[0031] The standard interactive dialogue, the standard scene image, and the standard scene data are fused to obtain multimodal fusion features.

[0032] A multimodal interaction device, comprising:

[0033] The current interaction script determination module is used to determine the current interaction script based on the current interaction voice at the moment of interaction;

[0034] The current scene information determination module is used to determine the current scene image and current scene data corresponding to the current interaction interface based on the current interaction interface corresponding to the interaction time.

[0035] The target control instruction determination module is used to perform multimodal recognition on the current interactive dialogue, the current scene image, and the current scene data using a multimodal recognition model to determine the target control instruction;

[0036] The target control instruction execution module is used to execute the target control instructions and complete the human-machine interaction operation.

[0037] Preferably, the target control command determination module includes:

[0038] The multimodal fusion feature determination submodule is used to determine multimodal fusion features based on the current interactive dialogue, the current scene image, and the current scene data;

[0039] The target control command determination submodule is used to perform intent recognition on the multimodal fusion features using an intent classification model to determine the target control command.

[0040] Preferably, the current scene data includes current scene text and current scene code;

[0041] The multimodal fusion feature determination submodule includes:

[0042] The first fusion feature determination unit is used to obtain a first fusion feature based on the current interactive dialogue, the current scene image and the current scene text, wherein the first fusion feature includes the location information of the required operation interface elements;

[0043] The second fusion feature determination unit is used to obtain a second fusion feature based on the current interactive dialogue and the current scene code. The second fusion feature includes code information of the required operation interface elements.

[0044] The multimodal fusion feature includes the first fusion feature and the second fusion feature.

[0045] Preferably, the current scene text includes the names and coordinates of all area blocks corresponding to the current interface elements;

[0046] The first fusion feature determination unit includes:

[0047] The first matching degree determination subunit is used to match the current interactive dialogue with the area block names corresponding to all current interface elements to determine the first matching degree corresponding to each current interface element.

[0048] The region block coordinate determination subunit is used to determine the current interface element as the required operation interface element if the first matching degree corresponding to the current interface element is greater than the first matching threshold, and to obtain the region block coordinates corresponding to the required operation interface element.

[0049] The coordinate regression processing subunit is used to perform coordinate regression processing on the region block coordinates corresponding to the required operation interface element and the current scene image to determine the first fusion feature.

[0050] Preferably, the current scene code includes code function comments corresponding to all current interface elements and scene code corresponding to the code function comments;

[0051] The second fusion feature determination unit includes:

[0052] The second matching degree determination subunit is used to match the current interactive dialogue with all code function comments corresponding to the current scene code, and determine the second matching degree corresponding to each code function comment;

[0053] The second fusion feature determination subunit is used to determine the scene code with the second matching degree greater than the second matching threshold as the second fusion feature.

[0054] Preferably, the target control command determination submodule includes:

[0055] The pending control instruction determination unit is used to perform intent recognition on the multimodal fusion features using an intent classification model, and to determine at least one pending control instruction corresponding to the required operation interface element and the recognition score corresponding to the pending control instruction.

[0056] The target control instruction determination unit is used to determine the pending control instruction with the highest identification score as the target control instruction.

[0057] Preferably, the multimodal fusion feature determination submodule includes:

[0058] The format conversion unit is used to convert the current interactive dialogue, the current scene image, and the current scene data into standard interactive dialogue, standard scene image, and standard scene data, respectively.

[0059] The feature fusion unit is used to perform feature fusion on the standard interactive dialogue, the standard scene image, and the standard scene data to obtain multimodal fusion features.

[0060] An in-vehicle controller includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned multimodal interaction method.

[0061] A vehicle infotainment system includes the aforementioned vehicle controller, a voice acquisition device connected to the vehicle controller, and a display screen; the voice acquisition device is used to send the acquired current interactive voice to the vehicle controller; the display screen is used to display the current scene image.

[0062] An automobile, including the aforementioned vehicle infotainment system.

[0063] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multimodal interaction method.

[0064] The aforementioned multimodal interaction method, device, controller, system, vehicle, and storage medium first acquire multimodal data reflecting the user's human-computer interaction process, such as the current interactive dialogue, current scene image, and current scene data corresponding to the same interaction moment. This multimodal data is then input into a pre-trained multimodal recognition model, which can directly output the required target control commands. This helps ensure the output efficiency and accuracy of the target control commands and reduces the complexity of control logic such as customized templates, rules, and associations during the development process, thereby improving the adaptability and generalization ability of voice interaction. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a schematic diagram of an application environment for a multimodal interaction method according to an embodiment of the present invention;

[0067] Figure 2 This is a flowchart of a multimodal interaction method according to an embodiment of the present invention;

[0068] Figure 3 This is another flowchart of a multimodal interaction method in one embodiment of the present invention;

[0069] Figure 4 This is another flowchart of a multimodal interaction method in one embodiment of the present invention;

[0070] Figure 5 This is another flowchart of a multimodal interaction method in one embodiment of the present invention;

[0071] Figure 6 This is another flowchart of a multimodal interaction method in one embodiment of the present invention;

[0072] Figure 7 This is another flowchart of a multimodal interaction method in one embodiment of the present invention;

[0073] Figure 8 This is a schematic diagram of a multimodal interaction device according to an embodiment of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] This invention provides a multimodal interaction method applicable to in-vehicle infotainment systems. The in-vehicle system includes an in-vehicle controller, a voice acquisition device communicatively connected to the in-vehicle controller, and a display screen. The in-vehicle controller is a controller installed in a vehicle; it can be a controller specifically designed for multimodal interaction or a controller integrating multimodal interaction functions and other functions. The voice acquisition device is used to acquire interactive voice data and can be, but is not limited to, a microphone. The display screen is used to display the interactive interface; it can be a touchscreen display that can both display the human-machine interface and receive control commands generated by user touch operations on the human-machine interface, executing corresponding control operations based on these commands.

[0076] In one embodiment, such as Figure 1 As shown, a multimodal interaction method is provided. Taking the application of this method in the vehicle controller of the in-vehicle infotainment system as an example, the method includes the following steps:

[0077] S101: Determine the current interaction script based on the current interaction voice at the moment of interaction;

[0078] S102: Based on the current interactive interface corresponding to the interaction time, determine the current scene image and current scene data corresponding to the current interactive interface;

[0079] S103: Employs a multimodal recognition model to perform multimodal recognition on the current interactive dialogue, current scene image, and current scene data to determine the target control command;

[0080] S104: Execute the target control command to complete the human-machine interaction operation.

[0081] Here, "current interactive speech" refers to the real-time voice collected for human-computer interaction. "Current interactive dialogue" refers to the user's speech script determined by speech recognition of the current interactive speech. "Interaction moment" refers to the moment the current interactive speech was collected.

[0082] In one embodiment, in step S101, the vehicle controller can receive the current interactive voice at the moment of interaction collected by a voice acquisition device (including but not limited to a microphone) installed in the vehicle. This current interactive voice can be understood as the voice collected by the user expressing their control intentions in a voice manner. After receiving the current interactive voice, the vehicle controller can use a pre-set voice recognition tool to perform voice recognition on the current interactive voice. For example, it can use an ASR recognition tool to recognize the current interactive voice and obtain the current interactive script in text form.

[0083] The current interactive interface refers to the interactive interface currently displayed on the vehicle's infotainment system screen. The current scene image refers to the interface image corresponding to the current interactive interface, which displays at least one current interface element. This current interface element is the interface element displayed in the current scene image; it can be understood as a control used to implement human-computer interaction, such as input boxes, buttons, scroll bars, and dropdown menus displayed on the current interactive interface. The current scene data refers to the tree structure corresponding to the current interactive interface. This tree structure includes nodes corresponding to the current interface elements, and each node includes scene text, scene location, node type, and other data.

[0084] As an example, in step 102, after receiving the current interactive voice, the vehicle controller can determine the interactive interface corresponding to the interaction time determined by the current interactive voice as the current interactive interface. That is, the current interactive interface can be understood as the interactive interface when the user expresses their current interactive voice. Then, through a pre-set interface, it obtains the current scene image and current scene data corresponding to the current interactive interface. For example, it can read the current scene image corresponding to the current interactive interface through a pre-set API interface and read the current scene data corresponding to the current interactive interface through a pre-set system service interface.

[0085] Among them, the multimodal recognition model is a model used to identify and process multimodal data in order to determine its control commands. Multimodal data here refers to data collected from different methods or perspectives for a given object, such as image data and text data representing the same object.

[0086] As an example, this multimodal recognition model is a recognition model pre-determined through model training using multimodal training samples. The multimodal training samples include multimodal training data and corresponding training control commands. The multimodal training data includes training interactive dialogue, training scene images, and training scene data. Here, the training interactive dialogue refers to the dialogue determined by recognizing interactive speech during model training; the training scene images refer to the scene images corresponding to the interactive interface determined during model training; the training scene data refers to the scene data corresponding to the interactive interface determined during model training; and the training control commands refer to the control commands that match the training interactive dialogue, training scene images, and training scene data. In this example, the training interactive dialogue, training scene images, training scene data, and training control commands are used as multimodal training samples input to a pre-set neural network model for model training. This allows the trained multimodal recognition model to output corresponding control commands based on the input multimodal data (including but not limited to interactive dialogue, scene images, and scene data).

[0087] For example, the multimodal training samples are as follows: query: training interaction script; sceneData: training scene data; Image: training scene image; Target: training control command; query: multimedia brightness adjustment to 60%, with a specific example as follows:

[0088]

[0089]

[0090] As an example, in step S103, the vehicle controller can input multimodal data such as the current interactive dialogue, the current scene image, and the current scene data into a pre-trained multimodal recognition model to quickly obtain the target control command output. Compared to the traditional method of simply using the current interactive dialogue to locate the scene information of the current interactive interface, using the current interactive dialogue combined with the current scene image and current scene data can quickly and accurately locate the focus interface element that the user needs to control, so as to output the target control command corresponding to the focus interface element, which helps to ensure the efficiency and accuracy of the target control command output. Compared to the traditional method that requires a large number of rules and templates for conversion control process, by recognizing multimodal data such as the current interactive dialogue, the current scene image, and the current scene data through a pre-trained multimodal recognition model, the required target control command can be directly output, reducing the complexity of custom templates, rules, and associations in the development process, which helps to improve the adaptability and generalization ability of voice interaction.

[0091] In this example, the identified target control instructions include: control type, control position, control ID, text, and action. Actions here include, but are not limited to, click, long press, setting value, setting swipe, and setting text. For example, in the target control instruction “rect:[308, 414, 338, 1073]|id:211|action:setValue|value:60%”, the control position is “[308, 414, 338, 1073]|”, the control ID is “id:211”, the text is “action:setValue”, and the action is “setting value:60%”.

[0092] As an example, in step S104, the vehicle controller can complete the human-machine interaction operation based on the target control command directly output by the multimodal recognition model.

[0093] In this embodiment, multimodal data that reflects the user's human-computer interaction process, such as the current interactive dialogue, the current scene image, and the current scene data corresponding to the same interaction moment, are first acquired. This multimodal data is then input into a pre-trained multimodal recognition model, which can directly output the required target control commands. This helps to ensure the output efficiency and accuracy of the target control commands, and can reduce the complexity of control logic such as customized templates, rules, and associations during the development process, thereby improving the adaptability and generalization ability of voice interaction.

[0094] In one embodiment, such as Figure 2 As shown, step S103 involves using a multimodal recognition model to perform multimodal recognition on the current interactive dialogue, the current scene image, and the current scene data to determine the target control command, including:

[0095] S201: Determine multimodal fusion features based on the current interactive dialogue, the current scene image, and the current scene data;

[0096] S202: Use an intent classification model to identify the intent of multimodal fusion features and determine the target control command.

[0097] As an example, the multimodal recognition model includes a feature fusion model and an intent classification model. The feature fusion model is used to perform feature extraction and fusion processing on the input multimodal data to output fused features corresponding to the multimodal data. The intent classification model is used to recognize the intent of the features corresponding to the multimodal data to output corresponding control commands.

[0098] As an example, in step S201, after acquiring multimodal data such as the current interactive dialogue, the current scene image, and the current scene data, the vehicle controller can use a pre-trained feature fusion model to extract and fuse features from the current interactive dialogue, the current scene image, and the current scene data. This allows for the rapid extraction of multimodal fusion features that reflect the user's intent. In this example, compared to the traditional method of simply using the current interactive dialogue to globally match all scene data of the vehicle system, using the current interactive dialogue combined with the current scene image and current scene data for local matching can quickly and accurately extract the multimodal fusion features corresponding to the current interactive dialogue.

[0099] As an example, in step S202, after acquiring multimodal fusion features extracted from multimodal data such as the current interactive dialogue, the current scene image, and the current scene data, the vehicle controller can input these multimodal fusion features into a pre-trained intent classification model for recognition, thereby obtaining the target control command output by the intent classification model. In this example, the intent classification model performs intent recognition on the multimodal fusion features, determines the target intent corresponding to the current interactive dialogue, and directly outputs the target control command corresponding to the target intent. Compared to the traditional method of first outputting the target intent and then using a large number of rules and templates for conversion control, this effectively reduces the complexity of customizing templates, rules, and associations during the development process, and helps improve the adaptability and generalization ability of voice interaction.

[0100] In this embodiment, features are first extracted and fused from multimodal data such as the current interactive dialogue, the current scene image, and the current scene data. This allows for the rapid and accurate acquisition of multimodal fusion features that represent the user's intent. Then, an intent classification model is used to identify the intent of the multimodal fusion features and directly output the target control command. This eliminates the need to first output the target intent and then use a large number of rules and templates for conversion. This effectively reduces the complexity of customizing templates, rules, and associations during development and helps improve the adaptability and generalization ability of voice interaction.

[0101] In one embodiment, the current scene data includes current scene text and current scene code;

[0102] like Figure 3 As shown, step S201 involves fusing features from the current interactive dialogue, the current scene image, and the current scene data to obtain multimodal fusion features, including:

[0103] S301: Based on the current interactive dialogue, the current scene image, and the current scene text, obtain the first fusion feature, which includes the location information of the required operation interface elements;

[0104] S302: Based on the current interaction dialogue and the current scene code, obtain the second fusion feature, which contains the code information of the required operation interface elements;

[0105] The multimodal fusion features include the first fusion feature and the second fusion feature.

[0106] The current scene text records the region block information corresponding to the currently displayed interface element in the current scene image. The current interface element here refers to the interface element displayed in the current scene image, which is the interface element that can interact with the user. Each region block information includes region block coordinates and region block name. The region block name is the name of the region block where a certain current interface element is located, recorded in the current scene data. The region block coordinates are the coordinates of the region block where a certain current interface element is located, recorded in the current scene data. For example, if the current scene image contains current interface elements corresponding to multiple menu areas such as "Audio," "Display," "System," "Basic Management," and "Application Management," then the current scene text extracted through the system service interface includes the region block information corresponding to the above menu areas, specifically including the region block coordinates and region block name.

[0107] The current scene code is used to record the code corresponding to the currently displayed interface element in the current scene image. For example, to implement multimedia brightness adjustment, in addition to displaying the current interface element for brightness adjustment on the current scene image for user touch operation, the current scene data will record the current scene code that implements this function. For example, the current scene code is: position: multimedia brightness adjustment | rect[624,278,414,237]-seekBar | rect[308, 414, 338, 1073]; type: TextView-SeekBar; slide: false-true; id: 209-211.

[0108] The required operation interface element refers to the current interface element that needs to be operated as indicated by the current interaction script.

[0109] As an example, in step S301, after acquiring multimodal data, the vehicle controller can perform feature matching on the current interactive dialogue, the current scene image, and the current scene text in the current scene data to determine the first fusion feature corresponding to the location information of the interface element required by the current interactive dialogue. This first fusion feature is the feature formed by fusing the above three modal data, specifically a feature formed by including the region block information corresponding to the interface element required by the current interactive dialogue, so as to quickly focus on and locate the required interface element. For example, the vehicle controller can perform matching processing on the current scene text corresponding to all current interface elements displayed in the current interactive dialogue and the current scene image to determine the region block information of the successfully matched current interface element in the current scene image, so as to quickly gather the required interface element. The required interface element here refers to the current interface element that needs to be operated as determined by the current interactive dialogue, such as the selected current interface element among multiple current interface elements such as "audio", "display", "system", "basic management" and "application management".

[0110] As an example, in step S302, after acquiring multimodal data, the vehicle controller can perform feature matching on the current interactive dialogue and the current scene code in the current scene data to determine the second fusion feature corresponding to the code information of the interface element required by the current interactive dialogue. This second fusion feature is the feature formed by fusing the two modal data, specifically the feature formed by the scene code corresponding to the interface element required by the current interactive dialogue. This allows for targeted determination of the scene code of the current interface element, providing a guarantee for subsequent intent recognition and direct output of the corresponding target control command. For example, the current scene code records the scene codes of all current interface elements displayed in the current scene image. The current interactive dialogue can be matched with the scene codes of all current interface elements to quickly determine the scene code corresponding to the required interface element, thereby obtaining the second fusion feature corresponding to the code information of the required interface element.

[0111] In this embodiment, based on the current interactive dialogue, the current scene image, and the current scene text, the first fusion feature corresponding to the location information of the required operation interface element can be quickly determined. Compared to the traditional method of simply using the current interactive dialogue to globally match all scene data of the vehicle system, the scene data to be matched determined based on the current scene image and the current scene text is less, and the first fusion feature corresponding to the location information of the required operation interface element can be quickly and accurately determined. Based on the current interactive dialogue and the current scene code, the second fusion feature corresponding to the code information of the required operation interface element can be quickly determined. Compared to the traditional method of simply using the current interactive dialogue to globally match all scene codes of the vehicle system layer by layer, the second fusion feature corresponding to the code information of the required operation interface element can be quickly and accurately determined because the scene code to be matched determined based on the current scene image is less.

[0112] In one embodiment, the current scene text includes the names and coordinates of all regions corresponding to the current interface elements;

[0113] like Figure 4 As shown, step S301, which involves obtaining the first fusion feature based on the current interactive dialogue, the current scene image, and the current scene text, includes:

[0114] S401: Match the current interactive dialogue with the names of all area blocks corresponding to the current interface elements to determine the first matching degree for each current interface element;

[0115] S402: If the first matching degree corresponding to the current interface element is greater than the first matching threshold, then the current interface element is determined as the required operation interface element, and the coordinates of the area block corresponding to the required operation interface element are obtained.

[0116] S403: Perform coordinate regression processing on the coordinates of the area block corresponding to the required operation interface element and the current scene image to determine the first fusion feature.

[0117] The first matching degree is the matching degree between the current interactive utterance and the area block name. The first matching threshold is a pre-set threshold used to evaluate whether a better matching standard has been met.

[0118] As an example, in step S401, the vehicle controller can match the current interactive dialogue with the names of all corresponding region blocks in the current scene text to determine the first matching degree for each current interface element. For example, the vehicle controller can use, but is not limited to, text matching algorithms such as Jaccard similarity, Levenshtein edit distance, Simhash, TF-IDF, and BM25 to match the current interactive dialogue with the names of all corresponding region blocks in the current interface to determine the first matching degree for each current interface element.

[0119] As an example, in step S402, after determining the first matching degree corresponding to each current interface element, the vehicle controller can compare the first matching degree with a preset first matching threshold. If the first matching degree is greater than the first matching threshold, it is determined that the area block name of the current interface element matches the current interactive dialogue to a high degree. The current interface element corresponding to the area block name can be determined as the required operation interface element, and then the area block coordinates corresponding to the required operation interface element can be determined.

[0120] As an example, in step S403, after determining the coordinates of the area block corresponding to the required operation interface element, the vehicle controller can use a pre-set coordinate regression algorithm to perform coordinate regression processing on the coordinates of the area block corresponding to the required operation interface element and the current scene image to obtain a first fusion feature containing the location information of the required operation interface element.

[0121] The vehicle controller can employ, but is not limited to, object detection algorithms such as Faster R-CNN, RetinaNet, and SSD to perform object detection on the current scene image and the coordinates of the regions corresponding to the required operation interface elements, thus determining a set of prediction boxes. For example, after determining the coordinates of the regions corresponding to the required operation interface elements, basic prediction box anchors can be generated based on these coordinates. The coordinates of these basic prediction box anchors are transformed into a center point, width *w*, and height *h*. The area of ​​the rectangle corresponding to the basic prediction box anchor is calculated using the width *w* and height *h*. Based on this rectangle area and a pre-set aspect ratio parameter *rations*, three first prediction box anchors are obtained. Then, all first prediction box anchors are expanded using a pre-set area ratio parameter *scale* to obtain a set of nine second prediction box anchors. Finally, the vehicle controller can employ, but is not limited to, a k-means clustering algorithm to perform coordinate regression processing on the prediction box set to obtain a first fusion feature containing the location information of the required operation interface elements. For example, first extract all bounding boxes (bboxes) corresponding to the current scene image; select the width (w) and height (h) of all bboxes; randomly select k values ​​of w and h from the 9 second predicted bounding box anchors as initial values; calculate the 9 Inversion of Values ​​(IOUs) corresponding to each bbox and the 9 second predicted bounding box anchors; determine the second predicted bounding box anchor corresponding to the minimum value among the 9 IOUs as the target anchor corresponding to that bbox, assign the bbox to the cluster of its corresponding target anchor, and repeat the above steps until its classification cluster does not change after updating the anchors, thus completing the coordinate regression operation, so that the finally determined second fusion feature can accurately locate the target predicted bounding box corresponding to the required operation interface element in the current scene image, and determine the position information corresponding to the target predicted bounding box as the first fusion feature.

[0122] In this embodiment, the current interactive dialogue is first matched with the region block names of all current interface elements corresponding to the current scene image to determine the required operation interface element with a high degree of matching. Since the number of current interface elements displayed in the current scene image is limited, the matching process is faster, which helps to improve the efficiency of obtaining the required operation interface element. Using the region block coordinates corresponding to the required operation interface element and the current scene image for coordinate regression processing can ensure the efficiency of obtaining the first fusion feature containing the location information of the required operation interface element, thereby improving the overall processing efficiency and positioning accuracy.

[0123] In one embodiment, the current scene code includes code function comments corresponding to all current interface elements and scene code corresponding to the code function comments;

[0124] like Figure 5As shown, step S302, which involves determining the second fusion feature based on the current interaction dialogue and the current scene code, includes:

[0125] S501: Match the current interactive dialogue with all code function comments corresponding to the current scenario code, and determine the second matching degree corresponding to each code function comment;

[0126] S502: The scene code with a second matching degree greater than the second matching threshold is identified as the second fusion feature.

[0127] Here, code function comments refer to text content that uses natural language to describe the function of the code content. Scenario code refers to code content used to implement a specific scenario function. The first matching degree is the degree of matching between the current interaction utterance and the code function comments. The second matching threshold is a pre-set threshold used to evaluate whether a better matching standard has been met.

[0128] As an example, in step S501, the vehicle controller can match the current interactive dialogue with all code function comments in the current scene code to determine the first matching degree corresponding to each code function comment. For example, the vehicle controller can use, but is not limited to, text matching algorithms such as Jaccard similarity, Levenshtein edit distance, Simhash, TF-IDF, and BM25 to match the current interactive dialogue with all code function comments to determine the second matching degree corresponding to each code function comment.

[0129] As an example, in step S502, after determining the second matching degree corresponding to each code function annotation, the vehicle controller can compare the second matching degree with a pre-set second matching threshold. If the second matching degree is greater than the second matching threshold, it is determined that the code function annotation matches the current interactive dialogue to a high degree. The scene code corresponding to the code function annotation can be determined as the code information corresponding to the required operation interface element to obtain the second fusion feature.

[0130] For example, the current scene code includes code block one and code block two. Code block one is "position: multimedia brightness adjustment|rect[624,278,414,237]-seekBar|rect[308,414,338,1073];

[0131] type:TextView-SeekBar; slide:false-true; id:209-211”, while the second code block is “position:Multimedia screensaver|rect[428,414,469,564]-clock1|rect[496,580,550,743];

[0132] "type:TextView-RadioButton; click:false-true; id:215-216". When the current interaction statement is "Adjust the multimedia brightness to 60%", the current interaction statement can be matched with code function annotations such as "Multimedia brightness adjustment" and "Multimedia screensaver" to determine the second matching degree A corresponding to "Multimedia brightness adjustment" and the second matching degree B corresponding to "Multimedia screensaver"; compare the second matching degree A and the second matching degree B with the second matching threshold S respectively. If A > S and B < S, then the code block one corresponding to the second matching degree A can be determined as the code information corresponding to the required operation interface element and determined as the second fusion feature.

[0133] In this embodiment, first, the current interaction statement is matched with the code function annotations in the current scene code to determine the scene code corresponding to the code function annotation with a higher matching degree, which is determined as the code information corresponding to the required operation interface element to obtain the second fusion feature. Since the number of current interface elements displayed in the current scene image is limited, the matching process is relatively fast, which helps to improve the acquisition efficiency of the required operation interface element, and further ensures the acquisition efficiency of the second fusion feature of the code information of the required operation interface element, thereby improving the overall processing efficiency.

[0134] In one embodiment, as Figure 6 shown, step S201, that is, performing feature fusion on the current interaction statement, the current scene image, and the current scene data to obtain a multi-modal fusion feature, includes:

[0135] S601: Perform format conversion on the current interaction statement, the current scene image, and the current scene data to obtain a standard interaction statement, a standard scene image, and standard scene data respectively;

[0136] S602: Perform feature fusion on the standard interaction statement, the standard scene image, and the standard scene data to obtain a multi-modal fusion feature.

[0137] Among them, the standard interaction statement is the statement determined after format conversion of the current interaction statement. The standard scene image is the scene image determined after format conversion of the current scene image. For example, it can be a scene image in RGB format. The standard scene data is the scene data determined after format conversion of the current scene data.

[0138] As an example, in S601, after the vehicle-mounted controller obtains multi-modal data, it can use the pre-set format conversion logic to perform format conversion on the current interaction statement, the current scene image, and the current scene data respectively to obtain the standard interaction statement, the standard scene image, and the standard scene data in the standard format, so that the format-converted multi-modal data is the same as the data format used in the multi-modal recognition model training process.

[0139] As an example, in step S602, after acquiring standard-format multimodal data such as standard interactive dialogue, standard scene images, and standard scene data, the vehicle controller can input this standard-format multimodal data into a pre-trained multimodal recognition model for multimodal recognition to determine the target control command. In this example, using a multimodal recognition model to recognize the standard-format multimodal data helps ensure the efficiency and accuracy of the output target control command.

[0140] In one embodiment, such as Figure 7 As shown, step S202 involves using an intent classification model to identify the intent of the multimodal fusion features and determine the target control command, including:

[0141] S701: The intent classification model is used to identify the intent of the multimodal fusion features and determine at least one pending control command and the recognition score corresponding to the required operation interface element.

[0142] S702: Identify the pending control command with the highest identification score as the target control command.

[0143] Among them, the pending control instructions refer to the control instructions determined by classifying the intent of multimodal fusion features using an intent classification model. The target control instruction is the control instruction with the highest recognition score among all pending control instructions.

[0144] As an example, in step S701, the vehicle controller uses an intent classification model to perform intent recognition on the multimodal fusion features formed by fusing the current interactive dialogue, the current scene image, and the current scene data. It can determine at least one pending control instruction corresponding to the required operation interface element displayed in the current scene image, and determine the recognition score corresponding to each pending control instruction. This recognition score can be understood as the probability that the pending control instruction is the actual control instruction corresponding to the current interactive dialogue.

[0145] As an example, in step S702, after determining the identification score corresponding to at least one pending control command, the vehicle controller can compare at least one identification score to determine the pending control command with the largest identification score as the target control command.

[0146] In this embodiment, an intent classification model is first used to identify the multimodal fusion features, determine the pending control instructions and their identification scores corresponding to each interface element in the current interactive interface, and determine the pending control instructions with the highest identification scores as the target control instructions corresponding to the current interactive dialogue. Since the number of current interface elements in the current scene image is limited, the identified pending control instructions are highly targeted, which helps to ensure the efficiency and accuracy of the final output target control instructions.

[0147] Compared to existing technologies that use interactive dialogue to globally match the text content of all scene data in the vehicle system and then determine the control command based on that text content, which suffers from low efficiency and accuracy, this solution integrates interactive dialogue, scene images, and scene data to achieve an effect similar to the coordinated action of the mouth and eyes. This has the following advantages: First, this solution can better understand user intent and directly target the required interface elements, helping to ensure processing efficiency and accuracy. Second, the multimodal recognition model in this solution directly outputs the target control command corresponding to the target intent. Compared to the traditional method of using numerous rules and templates to transform the target intent before determining the final target control command, this process is more precise, improving processing effectiveness. It also reduces the need for complex control logic such as customized templates, rules, and associations during development, enhancing the adaptability and generalization ability of voice interaction. Thirdly, since the multimodal recognition model is a model formed by pre-training, it has a certain generalization ability. It can be used for current scene images and current scene data with corresponding relationships. Compared with the existing technology, which requires the customization of rules and templates after interface changes due to system changes, third-party incoordination, or changes in system components, it has better universality. It can still be used directly after the vehicle system is upgraded and changed, which can greatly save product development and operation costs. It is not affected by the system model and can theoretically be used on Linux, Android, and IOS.

[0148] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0149] In one embodiment, a multimodal interaction device is provided, which corresponds one-to-one with the multimodal interaction methods described in the above embodiments. For example... Figure 8 As shown, the multimodal interaction device includes a current interaction script determination module 801, a current scene information determination module 802, a target control command determination module 803, and a target control command execution module 804. Detailed descriptions of each functional module are as follows:

[0150] The current interaction script determination module 801 is used to determine the current interaction script based on the current interaction voice at the moment of interaction;

[0151] The current scene information determination module 802 is used to determine the current scene image and current scene data corresponding to the current interaction interface based on the current interaction interface corresponding to the interaction time.

[0152] The target control instruction determination module 803 is used to perform multimodal recognition on the current interactive dialogue, the current scene image and the current scene data using a multimodal recognition model to determine the target control instruction;

[0153] The target control instruction execution module 804 is used to execute the target control instructions and complete the human-machine interaction operation.

[0154] In one embodiment, the target control command determination module 803 includes:

[0155] The multimodal fusion feature determination submodule is used to determine multimodal fusion features based on the current interactive dialogue, the current scene image, and the current scene data;

[0156] The target control command determination submodule is used to perform intent recognition on the multimodal fusion features using an intent classification model to determine the target control command.

[0157] In one embodiment, the current scene data includes current scene text and current scene code;

[0158] The multimodal fusion feature determination submodule includes:

[0159] The first fusion feature determination unit is used to obtain a first fusion feature based on the current interactive dialogue, the current scene image and the current scene text, wherein the first fusion feature includes the location information of the required operation interface elements;

[0160] The second fusion feature determination unit is used to obtain a second fusion feature based on the current interactive dialogue and the current scene code. The second fusion feature includes code information of the required operation interface elements.

[0161] The multimodal fusion feature includes the first fusion feature and the second fusion feature.

[0162] In one embodiment, the current scene text includes the names and coordinates of all current interface elements corresponding to their respective region blocks;

[0163] The first fusion feature determination unit includes:

[0164] The first matching degree determination subunit is used to match the current interactive dialogue with the area block names corresponding to all current interface elements to determine the first matching degree corresponding to each current interface element.

[0165] The region block coordinate determination subunit is used to determine the current interface element as the required operation interface element if the first matching degree corresponding to the current interface element is greater than the first matching threshold, and to obtain the region block coordinates corresponding to the required operation interface element.

[0166] The coordinate regression processing subunit is used to perform coordinate regression processing on the region block coordinates corresponding to the required operation interface element and the current scene image to determine the first fusion feature.

[0167] In one embodiment, the current scene code includes code function comments corresponding to all current interface elements and scene code corresponding to the code function comments;

[0168] The second fusion feature determination unit includes:

[0169] The second matching degree determination subunit is used to match the current interactive dialogue with all code function comments corresponding to the current scene code, and determine the second matching degree corresponding to each code function comment;

[0170] The second fusion feature determination subunit is used to determine the scene code with the second matching degree greater than the second matching threshold as the second fusion feature.

[0171] In one embodiment, the target control command determination submodule includes:

[0172] The pending control instruction determination unit is used to perform intent recognition on the multimodal fusion features using an intent classification model, and to determine at least one pending control instruction corresponding to the required operation interface element and the recognition score corresponding to the pending control instruction.

[0173] The target control instruction determination unit is used to determine the pending control instruction with the highest identification score as the target control instruction.

[0174] In one embodiment, the multimodal fusion feature determination submodule includes:

[0175] The format conversion unit is used to convert the current interactive dialogue, the current scene image, and the current scene data into standard interactive dialogue, standard scene image, and standard scene data, respectively.

[0176] The feature fusion unit is used to perform feature fusion on the standard interactive dialogue, the standard scene image, and the standard scene data to obtain multimodal fusion features.

[0177] For specific limitations regarding multimodal interaction devices, please refer to the limitations on multimodal interaction methods above, which will not be repeated here. Each module in the aforementioned multimodal interaction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0178] In one embodiment, an in-vehicle controller is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multimodal interaction method described in the above embodiments, for example... Figure 1 As shown in S101-S103, or Figures 2 to 7 As shown, to avoid repetition, it will not be described again here. Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in this embodiment of the multimodal interaction device, for example... Figure 8 The functions of the current interaction dialogue determination module 801, the current scene information determination module 802, the target control instruction determination module 803, and the target control instruction execution module 804 shown are not described again here to avoid repetition.

[0179] A vehicle-mounted infotainment system includes an in-vehicle controller, a voice acquisition device connected to the in-vehicle controller, and a display screen. The voice acquisition device transmits the acquired current interactive voice to the in-vehicle controller, and the display screen displays the current scene image. In this example, the voice acquisition device acquires the current interactive voice in real time and transmits it to the in-vehicle controller. After receiving the current interactive voice, the in-vehicle controller can obtain and read the current scene image and current scene data displayed on the display screen through a preset interface. Using a multimodal recognition model, the controller identifies the current interactive speech, current scene image, and current scene data corresponding to the current interactive voice, and can directly output a target control command. This allows for human-machine interaction based on the target control command, helping to ensure the output efficiency and accuracy of the target control command. Furthermore, it reduces the need for complex control logic such as customized templates, rules, and associations during development, improving the adaptability and generalization ability of voice interaction.

[0180] An automobile, including the aforementioned vehicle infotainment system.

[0181] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multimodal interaction method.

[0182] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the multimodal interaction method described in the above embodiments, for example... Figure 1 As shown in S101-S103, or Figures 2 to 7 As shown, to avoid repetition, it will not be described again here. Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in this embodiment of the multimodal interaction device, for example... Figure 8The functions of the current interaction dialogue determination module 801, the current scene information determination module 802, the target control instruction determination module 803, and the target control instruction execution module 804 shown are not described again here to avoid repetition.

[0183] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0184] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0185] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multimodal interaction method, characterized in that, include: Determine the current interaction script based on the current interaction voice at the moment of interaction; Based on the current interactive interface corresponding to the interaction time, determine the current scene image and current scene data corresponding to the current interactive interface; The current scene image refers to the interface image corresponding to the current interactive interface, and the interface image displays at least one current interface element. The current scene data refers to the tree structure corresponding to the current interactive interface. The tree structure includes the nodes corresponding to the current interface elements. The current scene data includes the current scene text and the current scene code. Based on the current interactive dialogue, the current scene image, and the current scene data, a multimodal fusion feature is determined; the multimodal fusion feature includes a first fusion feature and a second fusion feature, the first fusion feature containing the location information of the required operation interface elements, and the second fusion feature containing the code information of the required operation interface elements; An intent classification model is used to identify the intent of the multimodal fusion features and determine the target control command. Execute the target control command to complete the human-computer interaction operation.

2. The multimodal interaction method as described in claim 1, characterized in that, The step of fusing features from the current interactive dialogue, the current scene image, and the current scene data to obtain multimodal fusion features includes: Based on the current interactive dialogue, the current scene image, and the current scene text, obtain the first fusion feature; Based on the current interactive dialogue and the current scene code, obtain the second fusion feature.

3. The multimodal interaction method as described in claim 2, characterized in that, The current scene text includes the names and coordinates of all area blocks corresponding to the current interface elements; The step of obtaining the first fusion feature based on the current interactive dialogue, the current scene image, and the current scene text includes: The current interactive dialogue is matched with the area block names corresponding to all current interface elements to determine the first matching degree for each current interface element. If the first matching degree corresponding to the current interface element is greater than the first matching threshold, then the current interface element is determined as the required operation interface element, and the coordinates of the area block corresponding to the required operation interface element are obtained. Coordinate regression processing is performed on the region block coordinates corresponding to the required operation interface elements and the current scene image to determine the first fusion feature.

4. The multimodal interaction method as described in claim 2, characterized in that, The current scene code includes code function comments corresponding to all current interface elements and scene code corresponding to the code function comments; The step of determining the second fusion feature based on the current interaction dialogue and the current scene code includes: The current interactive dialogue is matched with all code function comments corresponding to the current scene code to determine the second matching degree corresponding to each code function comment; The scene codes with a second matching degree greater than the second matching threshold are identified as the second fusion feature.

5. The multimodal interaction method as described in claim 1, characterized in that, The step of using an intent classification model to identify the intent of the multimodal fusion features and determine the target control command includes: An intent classification model is used to identify the intent of the multimodal fusion features, and at least one pending control instruction corresponding to the required operation interface element and the identification score corresponding to the pending control instruction are determined. The control command with the highest identification score is identified as the target control command.

6. The multimodal interaction method as described in claim 1, characterized in that, The step of determining multimodal fusion features based on the current interactive dialogue, the current scene image, and the current scene data further includes: The current interactive dialogue, the current scene image, and the current scene data are converted into formats to obtain standard interactive dialogue, standard scene image, and standard scene data, respectively. The standard interactive dialogue, the standard scene image, and the standard scene data are fused to obtain multimodal fusion features.

7. A multimodal interaction device, characterized in that, include: The current interaction script determination module is used to determine the current interaction script based on the current interaction voice at the moment of interaction; The current scene information determination module is used to determine the current scene image and current scene data corresponding to the current interaction interface based on the current interaction interface corresponding to the interaction time. The current scene image refers to the interface image corresponding to the current interactive interface, and the interface image displays at least one current interface element. The current scene data refers to the tree structure corresponding to the current interactive interface. The tree structure includes the nodes corresponding to the current interface elements. The current scene data includes the current scene text and the current scene code. The multimodal fusion feature determination submodule is used to determine multimodal fusion features based on the current interactive dialogue, the current scene image, and the current scene data; the multimodal fusion features include a first fusion feature and a second fusion feature, wherein the first fusion feature contains the location information of the required operation interface elements, and the second fusion feature contains the code information of the required operation interface elements; The target control command determination submodule is used to perform intent recognition on the multimodal fusion features using an intent classification model to determine the target control command; The target control instruction execution module is used to execute the target control instructions and complete the human-machine interaction operation.

8. The multimodal interaction device as described in claim 7, characterized in that, The multimodal fusion feature determination submodule includes: The first fusion feature determination unit is used to obtain a first fusion feature based on the current interactive dialogue, the current scene image, and the current scene text; The second fusion feature determination unit is used to obtain the second fusion feature based on the current interactive dialogue and the current scene code.

9. The multimodal interaction device as described in claim 8, characterized in that, The current scene text includes the names and coordinates of all area blocks corresponding to the current interface elements; The first fusion feature determination unit includes: The first matching degree determination subunit is used to match the current interactive dialogue with the area block names corresponding to all current interface elements to determine the first matching degree corresponding to each current interface element. The region block coordinate determination subunit is used to determine the current interface element as the required operation interface element if the first matching degree corresponding to the current interface element is greater than the first matching threshold, and to obtain the region block coordinates corresponding to the required operation interface element. The coordinate regression processing subunit is used to perform coordinate regression processing on the region block coordinates corresponding to the required operation interface element and the current scene image to determine the first fusion feature.

10. The multimodal interaction device as described in claim 8, characterized in that, The current scene code includes code function comments corresponding to all current interface elements and scene code corresponding to the code function comments; The second fusion feature determination unit includes: The second matching degree determination subunit is used to match the current interactive dialogue with all code function comments corresponding to the current scene code, and determine the second matching degree corresponding to each code function comment; The second fusion feature determination subunit is used to determine the scene code with the second matching degree greater than the second matching threshold as the second fusion feature.

11. The multimodal interaction device as described in claim 7, characterized in that, The target control command determination submodule includes: The pending control instruction determination unit is used to perform intent recognition on the multimodal fusion features using an intent classification model, and to determine at least one pending control instruction corresponding to the required operation interface element and the recognition score corresponding to the pending control instruction. The target control instruction determination unit is used to determine the pending control instruction with the highest identification score as the target control instruction.

12. The multimodal interaction device as described in claim 7, characterized in that, The multimodal fusion feature determination submodule includes: The format conversion unit is used to convert the current interactive dialogue, the current scene image, and the current scene data into standard interactive dialogue, standard scene image, and standard scene data, respectively. The feature fusion unit is used to perform feature fusion on the standard interactive dialogue, the standard scene image, and the standard scene data to obtain multimodal fusion features.

13. An on-board controller, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal interaction method as described in any one of claims 1 to 6.

14. A vehicle infotainment system, characterized in that, The system includes the vehicle controller as described in claim 13, a voice acquisition device connected to the vehicle controller, and a display screen; the voice acquisition device is used to send the acquired current interactive voice to the vehicle controller; and the display screen is used to display the current scene image.

15. A car, characterized in that, Including the vehicle infotainment system as described in claim 14.

16. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal interaction method as described in any one of claims 1 to 6.

Images