Intelligent assistant interaction method, device and system based on multi-modal data fusion

Abstract

The invention relates to an intelligent assistant interaction method, device and system based on multi-modal data fusion, and the method comprises the following steps: 1) multi-modal perception: a user sends an instruction, and the system collects data through multi-modal perception; 2) multi-modal fusion: the system performs semantic fusion on the collected multi-modal data to generate task semantic representation; 3) task generation and execution: the system decomposes sub-tasks according to task types and distributes the sub-tasks to related equipment or services; 4) interactive feedback: the system feeds back an execution result to the user through voice broadcast or a display screen; and 5) learning and optimization: storing and learning a high-quality conclusion after interaction feedback, and enabling the execution of an obtained instruction to be more accurate through learning. According to the method and the system, multiple modes are combined, so that the system supports multiple input modes of voice, vision and texts, and the user interaction experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence, human-computer interaction and the Internet of Things, and specifically to an intelligent assistant interaction method, device and system based on multimodal data fusion. Background Art

[0002] With the development of artificial intelligence, existing technologies have been able to achieve intelligence through a single modality in specific scenarios. For example: 1. Smart home control based on voice recognition technology, which controls lighting, temperature and other devices through voice commands, but lacks the ability to understand complex tasks; 2. Security monitoring systems based on visual analysis can detect specific scenes or events, but they still have deficiencies in multimodal information fusion and interactivity; 3. Current task execution systems usually rely on pre-set logic and find it difficult to dynamically adjust tasks based on real-time environment and user intent.

[0003] However, these technologies have the following problems in practical applications: 1. Lack of multimodal fusion capabilities: The current system is unable to deeply integrate multimodal information such as voice, images, and text, resulting in inaccurate understanding.

[0004] 2. Single task processing: The system lacks task decomposition and collaboration capabilities, making it difficult to handle complex multi-tasking scenarios.

[0005] 3. The interaction is not natural enough: Users need to interact with the system in a specific way, and the system cannot be adaptively adjusted according to user behavior and environmental status.

[0006] To this end, an intelligent assistant interaction method, device and system based on multimodal data fusion are designed to overcome the above problems. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art and to provide an intelligent assistant interaction method, device and system based on multimodal data fusion, aiming to achieve comprehensive processing of natural language, vision, speech and environmental data for task collaboration and human-computer interaction in multiple scenarios.

[0008] The present invention is implemented by the following technical solution: an intelligent assistant interaction method based on multimodal data fusion, the method comprising the following steps: 1) Multimodal perception: The user issues a command, and the system collects data through multimodal perception; 2) Multimodal fusion: The system semantically fuses the collected multimodal data to generate task semantic representation; 3) Task generation and execution: The system decomposes subtasks according to task types and assigns them to relevant devices or services; 4) Interactive feedback: The system will feedback the execution results to the user through voice broadcast or display screen; 5) Learning and optimization: The high-quality conclusions after interactive feedback are saved and learned, and the execution of the instructions obtained through learning is made more accurate.

[0009] Preferably, the instructions issued by the user in step 1) include voice instructions, visual instructions, text instructions, and environmental perception instructions, wherein the voice instructions and visual instructions are identified, the text instructions are analyzed, and the environmental perception instructions are integrated after environmental collection.

[0010] As a preferred embodiment: the task generation and execution in step 3) are specifically divided into: A. Send the task to the execution device and interactive feedback at the same time; B. After receiving the task, the execution device will analyze and decompose the task C. After the decomposition is completed, the task equipment immediately starts task scheduling and execution, and after the execution is completed, it sends a completion signal to the interactive feedback D. Interactive feedback After receiving the task completion information, check it with the preset task information. If there is no preset, the latest processing result will be directly fed back. If there is a preset, start the comparison. If they match after comparison, feedback is given as a match. If they do not match, feedback is given as a problem. The final result can be manually set and used as a preset after input.

[0011] As a preferred embodiment, the specific method of learning and optimizing in step 5) is: saving the result in step 4), if it is consistent with the preset, skip saving, if it is inconsistent or there is no preset, directly save the final result, and repeat the learning operation.

[0012] An intelligent assistant interaction device based on multimodal data fusion, the device comprising: (1) Multimodal perception module: The speech perception unit collects user speech input and transcribes it into text; the visual perception unit collects environmental images through the camera and detects user gestures, expressions or objects; the text processing unit analyzes the text instructions input by the user; and the environmental perception unit collects environmental data.

[0013] (2) Multimodal fusion module: A deep learning-based fusion model uniformly encodes speech, vision, and text data; at the same time, it combines contextual information to generate multimodal semantic representations.

[0014] (3) Task processing and execution module: The task parsing unit generates a task plan based on the semantic representation; then the complex task is decomposed into subtasks through the task decomposition unit; after decomposition, the task scheduling unit is called to dynamically adjust the task execution order; finally, the device interface unit controls the smart device through the IoT protocol.

[0015] (4) Interactive feedback module: feedback on task execution status and results through voice broadcast, screen display or mobile terminal.

[0016] (5) Learning and optimization module: Dynamic knowledge base: stores user preferences, historical interaction records, and environment status; Reinforcement learning unit: Optimizing task allocation strategy based on feedback.

[0017] An assistant interaction system based on multimodal data fusion, which is used on a smart device. The system uses the above method or the above device to turn the device on and off by sensing voice commands, light commands, text commands, and visual commands.

[0018] Preferably, the smart device is used in smart home scenarios, smart office scenarios, and medical assistance scenarios.

[0019] The beneficial effects of the present invention are as follows: Compared with the existing intelligent assistant related technologies, the intelligent assistant interaction system based on multimodal data fusion provided by the present invention has the following beneficial effects: 1. Combining multimodality enables the system to support multiple input methods such as voice, vision and text, improving the user interaction experience.

[0020] 2. Optimize resource utilization and improve task execution efficiency through dynamic task decomposition and scheduling.

[0021] 3. The system can be adapted to various scenarios such as smart home, office automation, and medical assistance.

[0022] 4. Through reinforcement learning and knowledge base, the system can continuously optimize performance based on user behavior and environmental changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a system framework diagram of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to more clearly understand the objectives, technical solutions and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0025] In the description of the present invention, it is necessary to understand that the orientation or position relationship indicated by the terms such as "up", "down", "left", "right", "inside", "outside", "horizontal" and "vertical" is based on the orientation or position relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention, rather than indicating or implying that the device or original referred to must have a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0026] The present invention will be described in detail below with reference to the accompanying drawings: Figure 1 As shown, an intelligent assistant interaction method based on multimodal data fusion comprises the following steps: 1) Multimodal perception: The user issues a command, and the system collects data through multimodal perception; 2) Multimodal fusion: The system semantically fuses the collected multimodal data to generate task semantic representation; 3) Task generation and execution: The system decomposes subtasks according to task types and assigns them to relevant devices or services; 4) Interactive feedback: The system will feedback the execution results to the user through voice broadcast or display screen; 5) Learning and optimization: The high-quality conclusions after interactive feedback are saved and learned, and the execution of the instructions obtained through learning is made more accurate.

[0027] The instructions issued by the user in step 1) include voice instructions, visual instructions, text instructions, and environmental perception instructions, wherein the voice instructions and visual instructions are identified, the text instructions are analyzed, and the environmental perception instructions are integrated after environmental collection.

[0028] The task generation and execution in step 3) are specifically divided into: A. Send the task to the execution device and interactive feedback at the same time; B. After receiving the task, the execution device will analyze and decompose the task C. After the decomposition is completed, the task equipment immediately starts task scheduling and execution, and after the execution is completed, it sends a completion signal to the interactive feedback D. Interactive feedback After receiving the task completion information, check it with the preset task information. If there is no preset, the latest processing result will be directly fed back. If there is a preset, start the comparison. If they match after comparison, feedback is given as a match. If they do not match, feedback is given as a problem. The final result can be manually set and used as a preset after input.

[0029] The specific method of step 5) learning and optimization is: save the result in step 4), if it is consistent with the preset, skip saving, if it is inconsistent or there is no preset, directly save the final result, and repeat the learning operation.

[0030] An intelligent assistant interaction device based on multimodal data fusion, the device comprising: (1) Multimodal perception module: The speech perception unit collects user speech input and transcribes it into text; the visual perception unit collects environmental images through the camera and detects user gestures, expressions or objects; the text processing unit analyzes the text instructions input by the user; and the environmental perception unit collects environmental data.

[0031] (2) Multimodal fusion module: A deep learning-based fusion model uniformly encodes speech, vision, and text data; at the same time, it combines contextual information to generate multimodal semantic representations.

[0032] (3) Task processing and execution module: The task parsing unit generates a task plan based on the semantic representation; then the complex task is decomposed into subtasks through the task decomposition unit; after decomposition, the task scheduling unit is called to dynamically adjust the task execution order; finally, the device interface unit controls the smart device through the IoT protocol.

[0033] (4) Interactive feedback module: feedback on task execution status and results through voice broadcast, screen display or mobile terminal.

[0034] (5) Learning and optimization module: Dynamic knowledge base: stores user preferences, historical interaction records, and environment status; Reinforcement learning unit: Optimizing task allocation strategy based on feedback.

[0035] An assistant interaction system based on multimodal data fusion, which is used on a smart device. The system uses the above method or the above device to turn the device on and off by sensing voice commands, light commands, text commands, and visual commands.

[0036] The smart device is used in smart home scenarios, smart office scenarios, and medical assistance scenarios.

[0037] Example 1 Smart home scene User demand: The user issues a voice command to "turn on the lights, play music, and lower the room temperature."

[0038] System operation: Sense user voice commands and detect the ambient light in the living room. Decompose the task into three subtasks: "adjust the light", "start the speaker to play music", and "start the air conditioner to cool down". The system controls the light, sound and air conditioner equipment respectively through the IoT protocol. After the task is completed, the system voice feedback "The light is on, the music is playing, and the air conditioner has been adjusted to 22 degrees".

[0039] Example 2: Smart office scene User requirement: The user requests to "project the meeting schedule and send a notification" through a text command.

[0040] System Operation: The text processing unit interprets user commands while the visual module detects the projector status.

[0041] The task processing module generates three subtasks: "Start the projector", "Show the schedule", and "Send a notification via email". The system completes the schedule projection and email sending through the API. The system voice feedback "The meeting schedule has been projected and the notification has been sent".

[0042] Example 3: Medical assistance scenario User needs: Medical staff request patients' real-time health data through gesture commands.

[0043] System operation: The visual module recognizes the "health data" gesture and queries the patient's health database. The system generates a real-time health report based on the query results and presents it on the display. The voice module responds with "Health data has been updated and displayed."

[0044] Anyone familiar with the technology may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. An intelligent assistant interaction method based on multimodal data fusion, characterized in that: The method comprises the following steps: 1) Multimodal perception: The user issues a command, and the system collects data through multimodal perception; 2) Multimodal fusion: The system semantically fuses the collected multimodal data to generate task semantic representation; 3) Task generation and execution: The system decomposes subtasks according to task types and assigns them to relevant devices or services; 4) Interactive feedback: The system will feedback the execution results to the user through voice broadcast or display screen; 5) Learning and optimization: The high-quality conclusions after interactive feedback are saved and learned, and the execution of the instructions obtained through learning is made more accurate.

2. The intelligent assistant interaction method based on multimodal data fusion according to claim 1, characterized in that: The instructions issued by the user in step 1) include voice instructions, visual instructions, text instructions, and environmental perception instructions, wherein the voice instructions and visual instructions are identified, the text instructions are analyzed, and the environmental perception instructions are integrated after environmental collection.

3. The intelligent assistant interaction method based on multimodal data fusion according to claim 1, characterized in that: The task generation and execution in step 3) are specifically divided into: A. Send the task to the execution device and interactive feedback at the same time; B. After receiving the task, the execution device will analyze and decompose the task C. After the decomposition is completed, the task equipment immediately starts task scheduling and execution, and after the execution is completed, it sends a completion signal to the interactive feedback D. Interactive feedback After receiving the task completion information, check it with the preset task information. If there is no preset, the latest processing result will be directly fed back. If there is a preset, start the comparison. If they match after comparison, feedback is given as a match. If they do not match, feedback is given as a problem. The final result can be manually set and used as a preset after input.

4. The intelligent assistant interaction method based on multimodal data fusion according to claim 1, characterized in that: The specific method of step 5) learning and optimization is: save the result in step 4), if it is consistent with the preset, skip saving, if it is inconsistent or there is no preset, directly save the final result, and repeat the learning operation.

5. An intelligent assistant interaction device based on multimodal data fusion, characterized in that: The device comprises: (1) Multimodal perception module: The speech perception unit collects user speech input and transcribes it into text; the visual perception unit collects environmental images through the camera and detects user gestures, expressions or objects; the text processing unit analyzes the text instructions input by the user; and the environmental perception unit collects environmental data; (2) Multimodal fusion module: A deep learning-based fusion model uniformly encodes speech, vision, and text data; it also combines contextual information to generate multimodal semantic representations; (3) Task processing and execution module: The task parsing unit generates a task plan based on the semantic representation; then the complex task is decomposed into subtasks through the task decomposition unit; after decomposition, the task scheduling unit is called to dynamically adjust the task execution order; finally, the device interface unit controls the smart device through the IoT protocol; (4) Interactive feedback module: feedback on task execution status and results through voice broadcast, screen display or mobile terminal; (5) Learning and optimization module: Dynamic knowledge base: stores user preferences, historical interaction records, and environment status; Reinforcement learning unit: Optimizing task allocation strategy based on feedback.

6. An assistant interaction system based on multimodal data fusion, the system is used on a smart device, characterized in that: The system utilizes the method of any one of claims 1 to 4 or the device of claim 5 to turn the device on and off by sensing voice commands, light commands, text commands, and visual commands.

7. The assistant interaction system based on multimodal data fusion according to claim 6, characterized in that: The smart device is used in smart home scenarios, smart office scenarios, and medical assistance scenarios.

Images