Multi-modal interaction method and device

Abstract

The invention provides a multi-modal interaction method and device, and the method comprises the steps: 1, collecting target interaction data which comprises at least two of the eye movement, gesture, voice or physiological data of an interactor; 2, performing feature extraction on the target interaction data to generate feature interaction data; 3, fusing the feature interaction data according to a preset rule to generate a target feature vector; 4, identifying an interaction intention according to the target feature vector; 5, generating an interaction strategy according to the interaction intention and the driving scene; and step 6, executing an interaction instruction according to the interaction strategy. The intelligent level of interaction is improved, manual operation of an interactor is reduced, the operation requirement of the interactor is quickly responded, the distraction risk in the driving process is reduced, and the driving safety is improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a multimodal interaction method and device. Background Art

[0002] With the continuous development of automotive technology, a variety of interactive solutions are now available in the field of in-vehicle interaction, including touch interaction, voice interaction, gesture interaction, and interaction solutions that integrate multiple technologies. These current interaction methods all have limitations to varying degrees. For example, touch interaction requires manual operation by the driver, which can easily distract the driver; voice interaction is significantly affected by in-vehicle noise, resulting in unstable recognition accuracy; traditional eye movement and gesture interaction also face issues with accuracy and adaptability. Furthermore, existing multimodal fusion technologies are not intelligent enough in in-vehicle scenarios, making them difficult to flexibly adjust to complex and changing driving environments and personalized needs, thus failing to fully realize the advantages of multimodal interaction. Summary of the Invention

[0003] The present invention provides a multimodal interaction method and device, which aims to address the defects in the existing technology, improve the intelligence level of interaction, reduce the manual operation of the interactor, quickly respond to the operator's operation needs, reduce the risk of distraction during driving, and improve driving safety.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] One aspect of the present invention provides a multimodal interaction method, comprising:

[0006] Step 1: Collect target interaction data, where the target interaction data includes at least two of the interactor's eye movement, gesture, voice, or physiological data;

[0007] Step 2: extracting features from the target interaction data to generate feature interaction data;

[0008] Step 3: Fusing the feature interaction data according to preset rules to generate a target feature vector;

[0009] Step 4: identifying interaction intention based on the target feature vector;

[0010] Step 5: Generate an interaction strategy based on the interaction intention and driving scenario;

[0011] Step 6: Execute the interaction instruction according to the interaction strategy.

[0012] Specifically, the step 2 includes:

[0013] Step 21: pre-process the target interaction data, wherein the pre-processing includes filtering and denoising;

[0014] Step 22: Time-align the pre-processed target interaction data;

[0015] Step 23: Extract features from the aligned target interaction data.

[0016] Specifically, the preset rules include: fusing the eye movement, gesture, voice, and physiological data according to a first preset formula.

[0017] Specifically, the first preset formula is: I=W1*E+W2*G+W3*P+W4*V, where I is the fused target feature vector, E is the eye movement data feature vector, G is the gesture data feature vector, P is the physiological data feature vector, V is the voice data feature vector, W1, W2, W3, and W4 are the weights of eye movement, gesture, physiological, and voice data, respectively, and W1+W2+W3+W4=1.

[0018] Specifically, the weights W1, W2, W3, and W4 are based on a fuzzy logic algorithm and are adjusted in real time according to driving scenarios and physiological data.

[0019] Specifically, step 4 includes:

[0020] Step 41: Build an intent recognition model based on the Transformer architecture.

[0021] Step 42: Input the target feature vector and output the interaction intention through the multi-head attention mechanism.

[0022] Furthermore, after step 6, the method further includes:

[0023] Step 7: Collect the feedback results of the interaction instructions, adjust the interaction fusion strategy, and then return to step 3.

[0024] Another aspect of the present invention provides a multimodal interaction device, comprising: a multimodal data acquisition module, a data processing and fusion module, and an interactive decision and execution module connected in sequence;

[0025] The multimodal data acquisition module is used to collect target interaction data;

[0026] The data processing and fusion module is used to extract features from the target interaction data to generate feature interaction data, and fuse the feature interaction data according to preset rules to generate a target feature vector;

[0027] The interaction decision and execution module is used to identify the interaction intention based on the target feature vector, generate an interaction strategy based on the interaction intention and the driving scenario, and execute the interaction instruction according to the interaction strategy.

[0028] In this embodiment, the multimodal data acquisition module includes an eye movement data acquisition unit, a gesture data acquisition unit, a voice data acquisition unit, and a physiological data acquisition unit; the data processing and fusion module includes a data preprocessing unit, a feature extraction unit, and a dynamic weight fusion unit connected in sequence; the interactive decision and execution module includes an intention recognition unit, an interactive strategy generation unit, and an instruction execution unit connected in sequence;

[0029] The eye movement data collection unit is used to collect the eye movement data of the interactor;

[0030] The gesture data collection unit is used to collect gesture data of the interactor;

[0031] The voice data collection unit is used to collect the voice data of the interactor;

[0032] The physiological data collection unit is used to collect physiological data of the interactor;

[0033] The data preprocessing unit is used to preprocess and time-align the target interaction data;

[0034] The feature extraction unit is used to extract features from the aligned target interaction data;

[0035] The dynamic weight fusion unit is used to fuse the feature interaction data according to preset rules to generate a target feature vector;

[0036] The intention recognition unit is used to recognize the interaction intention according to the target feature vector;

[0037] The interaction strategy generating unit is used to generate an interaction strategy according to the interaction intention and the driving scenario;

[0038] The instruction execution unit is used to execute the interaction instruction according to the interaction strategy.

[0039] Furthermore, the interactive decision and execution module also includes a feedback collection unit connected to the instruction execution unit, and the feedback collection unit is also connected to the dynamic weight fusion unit; the feedback collection unit is used to collect feedback results of interactive instructions and send them to the dynamic weight fusion unit for interactive fusion strategy adjustment.

[0040] The beneficial effects of the present invention are as follows: the present invention dynamically integrates the eye movements, gestures, voice and physiological data of the interactors, dynamically adjusts the fusion weight according to the driving scenario and the interactor's status, and collects and analyzes the interaction data to learn the interactors' usage habits and preferences, further optimizes the fusion weight and interaction strategy, and more accurately understands the interactors' operational intentions, thereby improving the intelligent level of the interaction, reducing the interactors' manual operations, quickly responding to the interactors' operational needs, reducing the risk of distraction during driving, and improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the multimodal interaction method of the present invention;

[0042] Figure 2 is a schematic structural diagram of a multimodal interaction device of the present invention;

[0043] Figure 3 It is another structural schematic diagram of the multimodal interaction device of the present invention. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings, which are for reference and illustration only and do not limit the scope of patent protection of the present invention.

[0045] In the processes described in the specification, claims, or drawings of the present invention, if the steps are numbered (e.g., steps 10, 20, etc.), the numbers are used solely to distinguish the steps and do not represent any order of execution. It should be noted that the terms "first," "second," etc., used herein are used solely to distinguish the objects being described and do not indicate a sequential order or indicate different types of steps.

[0046] Example 1

[0047] like Figure 1 As shown, this embodiment provides a multimodal interaction method, including:

[0048] Step 1: Collect target interaction data, where the target interaction data includes at least two of the interactor's eye movements, gestures, voice, or physiological data.

[0049] In specific implementation, the eye movement data includes information such as gaze point coordinates, blinking frequency, pupil size, etc.; the gesture data includes information such as hand shape, posture, position, movement trajectory, etc.; the voice data includes voice recognition results; and the physiological data includes information such as heart rate, breathing rate, and body pressure distribution.

[0050] Step 2: Extract features from the target interaction data to generate feature interaction data.

[0051] In this embodiment, step 2 includes:

[0052] Step 21: pre-process the target interaction data, wherein the pre-processing includes filtering and denoising.

[0053] Step 22: Time-align the pre-processed target interaction data.

[0054] Step 23: Extract features from the aligned target interaction data.

[0055] In specific implementation, feature extraction is performed through deep learning and machine learning algorithms.

[0056] For eye movement data, features such as the speed, acceleration, and duration of gaze are extracted; for gesture data, features such as changes in hand joint angles, gesture amplitude, and speed are extracted; for speech data, features such as frequency, intonation, speaking speed, and key semantic features are extracted; for physiological data, features such as heart rate variability and changes in breathing rhythm are extracted.

[0057] Step 3: Fusing the feature interaction data according to preset rules to generate a target feature vector.

[0058] In this embodiment, the preset rule includes: fusing the eye movement, gesture, voice, and physiological data according to a first preset formula.

[0059] In this embodiment, the first preset formula is: I=W1*E+W2*G+W3*P+W4*V, where I is the fused target feature vector, E is the eye movement data feature vector, G is the gesture data feature vector, P is the physiological data feature vector, V is the voice data feature vector, W1, W2, W3, and W4 are the weights of eye movement, gesture, physiological, and voice data, respectively, and W1+W2+W3+W4=1.

[0060] In this embodiment, the weights W1, W2, W3, and W4 are based on a fuzzy logic algorithm and are adjusted in real time according to driving scenarios and physiological data.

[0061] For example, in a highway driving scenario, if the driver's heart rate increases and his eye movements are frequent, it means that he may be in a nervous state. At this time, the weight of the eye movement data can be increased to quickly respond to the user's intention; in a parking scenario, gesture operation is more convenient, and the weight of gesture data can be increased; when the in-car environment is relatively quiet, voice commands are more reliable, and the weight of voice data can be increased.

[0062] Step 4: Identify the interaction intention based on the target feature vector.

[0063] In this embodiment, step 4 includes:

[0064] Step 41: Build an intent recognition model based on the Transformer architecture.

[0065] Step 42: Input the target feature vector and output the interaction intention through the multi-head attention mechanism.

[0066] In specific implementation, the intention recognition model is trained with a large amount of actual driving scenario data and can accurately identify common interaction intentions, such as setting navigation destinations, controlling music playback, opening and closing windows, etc.

[0067] Step 5: Generate an interaction strategy based on the interaction intention and driving scenario.

[0068] During specific implementation, the driving scene can be set according to the needs of the interactor, such as a high-speed driving scene, a parking scene, a reversing scene, etc.

[0069] For example, when it is recognized that the driver intends to adjust the air conditioning temperature, if the vehicle is in a high-speed driving scenario, the operation will be confirmed by voice broadcast to avoid distraction of the driver; if it is in a parking scenario, a simple air conditioning adjustment interface will pop up, supporting quick adjustment by gestures.

[0070] Step 6: Execute the interaction instruction according to the interaction strategy.

[0071] During specific implementation, the interaction strategy is converted into specific control instructions and sent to various functional modules of the vehicle-mounted system for execution.

[0072] In another embodiment of the present invention, after step 6, the method further includes:

[0073] Step 7: Collect the feedback results of the interaction instructions, adjust the interaction fusion strategy, and then return to step 3.

[0074] This step forms a closed-loop control by collecting the feedback execution results of the interactive instructions so that the system can further optimize the subsequent interactive process based on the execution effect.

[0075] Example 2

[0076] like Figure 2 As shown, this embodiment provides a multimodal interaction device, including a multimodal data acquisition module, a data processing and fusion module, and an interactive decision and execution module connected in sequence;

[0077] The multimodal data acquisition module is used to collect target interaction data;

[0078] The data processing and fusion module is used to extract features from the target interaction data to generate feature interaction data, and fuse the feature interaction data according to preset rules to generate a target feature vector;

[0079] The interaction decision and execution module is used to identify the interaction intention based on the target feature vector, generate an interaction strategy based on the interaction intention and the driving scenario, and execute the interaction instruction according to the interaction strategy.

[0080] In this embodiment, the multimodal data acquisition module includes an eye movement data acquisition unit, a gesture data acquisition unit, a voice data acquisition unit, and a physiological data acquisition unit; the data processing and fusion module includes a data preprocessing unit, a feature extraction unit, and a dynamic weight fusion unit connected in sequence; the interactive decision and execution module includes an intention recognition unit, an interactive strategy generation unit, and an instruction execution unit connected in sequence;

[0081] The eye movement data collection unit is used to collect the eye movement data of the interactor;

[0082] The gesture data collection unit is used to collect gesture data of the interactor;

[0083] The voice data collection unit is used to collect the voice data of the interactor;

[0084] The physiological data collection unit is used to collect physiological data of the interactor;

[0085] The data preprocessing unit is used to preprocess and time-align the target interaction data;

[0086] The feature extraction unit is used to extract features from the aligned target interaction data;

[0087] The dynamic weight fusion unit is used to fuse the feature interaction data according to preset rules to generate a target feature vector;

[0088] The intention recognition unit is used to recognize the interaction intention according to the target feature vector;

[0089] The interaction strategy generating unit is used to generate an interaction strategy according to the interaction intention and the driving scenario;

[0090] The instruction execution unit is used to execute the interaction instruction according to the interaction strategy.

[0091] like Figure 3 As shown, in another embodiment of the present invention, the interactive decision and execution module also includes a feedback acquisition unit connected to the instruction execution unit, and the feedback acquisition unit is also connected to the dynamic weight fusion unit; the feedback acquisition unit is used to collect feedback results of the interactive instructions and send them to the dynamic weight fusion unit for interactive fusion strategy adjustment.

[0092] The working process of the multimodal interaction device in this embodiment is as described in the multimodal interaction method in Example 1, and will not be repeated here.

[0093] The above disclosure is only a preferred embodiment of the present invention and cannot be used to limit the scope of protection of the present invention. Therefore, equivalent changes made according to the scope of the patent application of the present invention are still within the scope covered by the present invention.

Claims

1. A multimodal interaction method, characterized in that: include: Step 1: Collect target interaction data, where the target interaction data includes at least two of the interactor's eye movement, gesture, voice, or physiological data; Step 2: extracting features from the target interaction data to generate feature interaction data; Step 3: Fusing the feature interaction data according to preset rules to generate a target feature vector; Step 4: identifying interaction intention based on the target feature vector; Step 5: Generate an interaction strategy based on the interaction intention and driving scenario; Step 6: Execute the interaction instruction according to the interaction strategy.

2. The multimodal interaction method according to claim 1, characterized in that: The step 2 includes: Step 21: pre-process the target interaction data, wherein the pre-processing includes filtering and denoising; Step 22: Time-align the pre-processed target interaction data; Step 23: Extract features from the aligned target interaction data.

3. The multimodal interaction method according to claim 2, wherein: The preset rules include: fusing the eye movement, gesture, voice, and physiological data according to a first preset formula.

4. The multimodal interaction method according to claim 3, wherein: The first preset formula is: I=W1*E+W2*G+W3*P+W4*V, where I is the fused target feature vector, E is the eye movement data feature vector, G is the gesture data feature vector, P is the physiological data feature vector, V is the voice data feature vector, W1, W2, W3, and W4 are the weights of eye movement, gesture, physiological, and voice data, respectively, and W1+W2+W3+W4=1.

5. The multimodal interaction method according to claim 4, characterized in that: The weights W1, W2, W3, and W4 are based on a fuzzy logic algorithm and are adjusted in real time according to driving scenarios and physiological data.

6. The multimodal interaction method according to claim 5, characterized in that: The step 4 comprises: Step 41: Build an intent recognition model based on the Transformer architecture. Step 42: Input the target feature vector and output the interaction intention through the multi-head attention mechanism.

7. The multimodal interaction method according to claim 1, characterized in that: After step 6, the method further includes: step 7, collecting feedback results of the interaction instructions, adjusting the interaction fusion strategy, and returning to step 3.

8. A multimodal interaction device, characterized in that: include: The multimodal data acquisition module, data processing and fusion module, and interactive decision-making and execution module are connected in sequence; The multimodal data acquisition module is used to collect target interaction data; The data processing and fusion module is used to extract features from the target interaction data to generate feature interaction data, and fuse the feature interaction data according to preset rules to generate a target feature vector; The interaction decision and execution module is used to identify the interaction intention based on the target feature vector, generate an interaction strategy based on the interaction intention and the driving scenario, and execute the interaction instruction according to the interaction strategy. In this embodiment, the multimodal data acquisition module includes an eye movement data acquisition unit, a gesture data acquisition unit, a voice data acquisition unit, and a physiological data acquisition unit; the data processing and fusion module includes a data preprocessing unit, a feature extraction unit, and a dynamic weight fusion unit connected in sequence; the interactive decision and execution module includes an intention recognition unit, an interactive strategy generation unit, and an instruction execution unit connected in sequence; The eye movement data collection unit is used to collect the eye movement data of the interactor; The gesture data collection unit is used to collect gesture data of the interactor; The voice data collection unit is used to collect the voice data of the interactor; The physiological data collection unit is used to collect physiological data of the interactor; The data preprocessing unit is used to preprocess and time-align the target interaction data; The feature extraction unit is used to extract features from the aligned target interaction data; The dynamic weight fusion unit is used to fuse the feature interaction data according to preset rules to generate a target feature vector; The intention recognition unit is used to recognize the interaction intention according to the target feature vector; The interaction strategy generating unit is used to generate an interaction strategy according to the interaction intention and the driving scenario; The instruction execution unit is used to execute the interaction instruction according to the interaction strategy.

9. The multimodal interaction device according to claim 8, characterized in that: The interactive decision and execution module also includes a feedback collection unit connected to the instruction execution unit, and the feedback collection unit is also connected to the dynamic weight fusion unit; the feedback collection unit is used to collect feedback results of interactive instructions and send them to the dynamic weight fusion unit for interactive fusion strategy adjustment.

Images