An intelligent space solution based on agent and multimodal understanding

Abstract

An intelligent space solution based on agent and multimodal understanding, with a large language model as the core of the intelligent agent, combined with the multimodal large model to perform comprehensive analysis on images and texts, wherein the large language model and the multimodal large model can be directly used as open source or fine-tuned for spatial scenes. The invention distributes user instructions to corresponding functional modules through the large language model agent, and completes complex tasks in combination with the multimodal large model, which significantly improves the efficiency, accuracy and flexibility of task processing, supports multi-functional collaboration such as spatial scene understanding, target distance estimation, item statistics, behavior recognition, etc., and meets the application needs of multiple scenes.

Description

Technical Field

[0001] The present invention relates to the fields of smart space and artificial intelligence, and in particular to an intelligent space assistant based on Agent and multimodal understanding. Background Art

[0002] At present, research and implementation of smart space assistants have made some progress. For example, rule-based automation systems can achieve simple equipment control, space monitoring and alarm functions; smart home systems use sensors and cameras for scene monitoring, switching equipment and intrusion detection. In addition, computer vision-based systems can achieve object detection, behavior recognition and scene analysis through cameras, and support some basic tasks such as counting the number of objects and detecting abnormal behavior. However, these technologies have the following problems and shortcomings: 1. Functional modules are independent and lack integration: Existing systems are often designed for a single function. For example, the device control system and behavior monitoring system are implemented independently and lack unified scheduling and collaboration.

[0003] 2. Difficulty in processing complex instructions and multimodal information: Many systems only support the processing of a single modality (such as images or voice), and the recognition and matching rules for instructions are relatively fixed, making it difficult to achieve cross-modal understanding and comprehensive reasoning, resulting in limited support for complex scenarios and instructions.

[0004] 3. Lack of flexibility and scalability: Existing technologies rely on fixed rules or training data. For example, target detection can only achieve the targets included in the training set, and it is difficult to flexibly adjust according to changes in scenarios or user needs.

[0005] 4. Lack of initiative and real-time performance: Many systems only respond passively when the user gives clear instructions, and have limited real-time and proactive warning capabilities in anomaly detection or scene monitoring. Summary of the invention

[0006] The purpose of the present invention is to overcome the above-mentioned deficiencies in the prior art and propose an intelligent space assistant based on agent and multimodal understanding, which can efficiently integrate and dispatch multiple cameras to realize functions such as spatial scene understanding, object quantity counting, target distance estimation, behavior understanding, equipment control and abnormal warning. It uses a large language model to perform flexible agent and utilizes a large multimodal model to improve the understanding and response capabilities of complex scenes and instructions, providing an efficient, flexible and real-time intelligent space management solution.

[0007] An intelligent space solution based on agent and multimodal understanding, the method uses a large language model as the core of the intelligent agent, combined with the multimodal large model to perform comprehensive analysis on images and texts, wherein the large language model and the multimodal large model can be directly used as open source or fine-tuned for spatial scenes, the method specifically includes: 1) System wake-up: The user speaks a specific wake-up word to the system through the microphone. The system monitors and recognizes the wake-up word in real time and activates the service process of the smart space assistant. After wake-up, the system confirms the activation status through voice prompts or light indications, and prompts the user to enter instructions; 2) Command input and analysis: The user inputs the task command in natural language, and the system converts the voice command into text through the speech recognition module and inputs it into the large language model agent; 3) Camera scene acquisition: The Agent instructs the camera in the corresponding area to capture real-time scene images or call historical videos according to task requirements. During the image capture process, the system automatically matches the area name or number in the instruction with the coverage of the camera to ensure that the data source is accurate; 4) Functional module pre-definition and design: support multiple functional modules and sustainable expansion; 5) Multimodal understanding and task execution: The captured scene images and parsed text instructions are input into the multimodal large model. The multimodal large model integrates visual information and text information to complete the understanding of the scene and the reasoning and execution of the task; 6) Result processing and feedback: After the multimodal large model completes the task, the feedback content is finally converted into voice broadcast to the user through the speech synthesis module, and it can also be displayed in text form on the visual interface; 7) Task recording and optimization: The system will record the task process and results in the background database for subsequent analysis and performance optimization.

[0008] As a preferred embodiment: the specific process inside the language model Agent in step 2) is: 1) Semantic parsing: The large language model parses the instruction text to extract key intent, parameters, and task content; 2) Task allocation: Based on the analysis results, the Agent allocates the task to the corresponding functional module and selects the camera or device to be called according to the task type.

[0009] As a preferred embodiment: the functional modules in step 4) are as follows: 1) Spatial scene understanding module: supports single-region or multi-region scene analysis, assigns tasks to multimodal large models through agents, and outputs scene description results or task conclusions; 2) Target quantity statistics module: Counts the number of targets by region, supports users to specify regions in instructions, and calculates results through multi-modal large model analysis; 3) Target distance estimation module: Calculates the distance between two targets through depth estimation algorithm combined with multimodal large model understanding and positioning, which is suitable for measurement tasks within the coverage area of ​​a single camera; 4) Behavior understanding module: Analyze the behavior of designated personnel in a single area or multiple areas, such as identifying the behavior type and generating the corresponding behavior description; 5) Equipment control module: directly controls the on / off status of the equipment based on user instructions without the need for area designation; 6) Abnormal warning module: Actively or passively detect abnormal situations in the entire area. Passive detection is carried out by a background process that monitors the entire area once every 10 seconds. The camera monitors the entire area. The multimodal large model understands possible abnormal behaviors in the scene and issues alarms, such as fights between people, abnormal fireworks, etc., to achieve safety management through real-time monitoring and alarm notifications.

[0010] Compared with the existing intelligent space assistants, the intelligent space assistant based on agent and multimodal understanding provided by the present invention has the following beneficial effects: 1. Multi-task collaborative execution: The present invention distributes user instructions to corresponding functional modules through a large language model Agent, and completes complex tasks in combination with a multimodal large model, significantly improving the efficiency, accuracy and flexibility of task processing, supporting multi-functional collaboration such as spatial scene understanding, target distance estimation, object statistics, behavior recognition, etc., and meeting the application needs of multiple scenarios.

[0011] 2. Easy operation and human-computer friendly interaction: The system supports the input of task instructions through natural voice interaction. Taking advantage of the large language model, it does not require completely solidified instructions or complicated manual operations, which greatly reduces the threshold for use. At the same time, the results are fed back in the form of voice or text, further optimizing the user experience.

[0012] 3. Strong versatility: Combining advanced depth estimation, large language model and multimodal large model technology, the system can achieve general scene understanding and task reasoning, avoiding the limitations of traditional natural language understanding and target detection solutions.

[0013] 4. Efficient resource utilization: The system uses a multi-task modular design to call resources on demand, avoiding resource waste. In the abnormal warning module, a mechanism combining timed detection and real-time events is adopted to balance performance and energy consumption requirements and save computing resources.

[0014] 5. Scalability and adaptability: The system supports on-demand expansion of new functional modules, such as adding specific device control logic or specific anomaly detection types; regional divisions and functional parameters can be flexibly configured to adapt to different application scenarios (such as office buildings, factories or home environments).

[0015] 6. Environmental and safety benefits: The system can effectively monitor abnormal situations in the environment (such as fire or fights), issue alarms in time, and ensure space safety; through intelligent control of equipment (such as air conditioning and lighting), it reduces energy consumption and achieves green energy-saving effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be described in detail below with reference to the accompanying drawings: Figure 1 As shown, an intelligent space solution based on agent and multimodal understanding is provided. The method uses a large language model as the core intelligent agent and combines the multimodal large model to perform comprehensive analysis on images and texts. The large language model and the multimodal large model can be directly used as open source or fine-tuned for spatial scenes. The method specifically includes: 1) System wake-up: The user speaks a specific wake-up word to the system through the microphone. The system monitors and recognizes the wake-up word in real time and activates the service process of the smart space assistant. After wake-up, the system confirms the activation status through voice prompts or light indications, and prompts the user to enter instructions; 2) Command input and analysis: The user inputs the task command in natural language, and the system converts the voice command into text through the speech recognition module and inputs it into the large language model agent; 3) Camera scene acquisition: The Agent instructs the camera in the corresponding area to capture real-time scene images or call historical videos according to task requirements. During the image capture process, the system automatically matches the area name or number in the instruction with the coverage of the camera to ensure that the data source is accurate; 4) Functional module pre-definition and design: support multiple functional modules and sustainable expansion; 5) Multimodal understanding and task execution: The captured scene images and parsed text instructions are input into the multimodal large model. The multimodal large model integrates visual information and text information to complete the understanding of the scene and the reasoning and execution of the task; 6) Result processing and feedback: After the multimodal large model completes the task, the feedback content is finally converted into voice broadcast to the user through the speech synthesis module, and it can also be displayed in text form on the visual interface; 7) Task recording and optimization: The system will record the task process and results in the background database for subsequent analysis and performance optimization.

[0018] The specific process inside the language model Agent is as follows: 1) Semantic parsing: The large language model parses the instruction text to extract key intent, parameters, and task content; 2) Task allocation: Based on the analysis results, the Agent allocates the task to the corresponding functional module and selects the camera or device to be called according to the task type.

[0019] The functional modules are as follows: 1) Spatial scene understanding module: supports single-region or multi-region scene analysis, assigns tasks to multimodal large models through agents, and outputs scene description results or task conclusions; 2) Target quantity statistics module: Counts the number of targets by region, supports users to specify regions in instructions, and calculates results through multi-modal large model analysis; 3) Target distance estimation module: Calculates the distance between two targets through depth estimation algorithm combined with multimodal large model understanding and positioning, which is suitable for measurement tasks within the coverage area of ​​a single camera; 4) Behavior understanding module: Analyze the behavior of designated personnel in a single area or multiple areas, such as identifying the behavior type and generating the corresponding behavior description; 5) Equipment control module: directly controls the on / off status of the equipment based on user instructions without the need for area designation; 6) Abnormal warning module: Actively or passively detect abnormal situations in the entire area. Passive detection is carried out by a background process that monitors the entire area once every 10 seconds. The camera monitors the entire area. The multimodal large model understands possible abnormal behaviors in the scene and issues alarms, such as fights between people, abnormal fireworks, etc., to achieve safety management through real-time monitoring and alarm notifications.

[0020] The present invention is based on agent and multimodal understanding technology. The user activates the system with a specific wake-up word through a microphone, and then inputs a natural language instruction; the system converts the instruction into text through voice recognition and inputs it into the large language model Agent; after the Agent parses the instruction, it is assigned to the corresponding functional module, and the scene image is collected through the camera, and the multimodal large model is called to complete the task; the final result is fed back to the user in the form of voice or text.

[0021] After specific implementation, the usage examples are as follows: Example 1: Spatial scene understanding The user uses the voice command "Help me analyze the overall scene of a certain area". After waking up, the system calls the camera of the corresponding area, captures the scene image and inputs it into the multimodal large model. The model outputs information including the distribution of people in the area, the status of office equipment in use, etc. Finally, the system gives feedback through voice, such as "There are 5 people in a certain area, they are all working, and all monitors are on."

[0022] Example 2: Counting the number of items The user instructs "to count the number of displays in the entire area". After the system wakes up, it parses the instruction and calls the corresponding camera to capture images, and inputs the images into the multimodal large model to analyze the number of displays. The system combines the scene partition number, summarizes the statistical results of multiple cameras, and then feeds back, for example, "there are 12 displays in the entire area".

[0023] Example 3: Target Distance Estimation The user instructs "estimate the distance between two white tables in a certain area". After the system is activated, it interprets the instruction and calls the camera in the corresponding area to capture the image and input it into the multimodal model for target positioning and distance calculation. Combined with the depth estimation algorithm, the system gives feedback, for example, "the distance between two white tables in a certain area is about 1.5 meters".

[0024] Example 4: Behavioral Understanding The user instructs "check the behavior of people in a certain area". After the system is activated, it captures the real-time image of the conference room and inputs it into the multimodal model to analyze the behavior of the people and provide feedback, such as "there are 6 people in a certain area, 4 of whom are discussing work, and 2 are using their mobile phones."

[0025] Example 5: Device Control The user instructs "turn off all lights in the entire area". After parsing the instruction, the system directly calls the device control module, sends the turn-off instruction through the IoT gateway, successfully turns off the lights and gives feedback, such as "all lights in the entire area have been turned off".

[0026] Example 6: Abnormal situation warning The system background process is set to perform anomaly detection on the monitored area every 10 seconds. During operation, a fight broke out in a certain area. The system identified the abnormal behavior through a multimodal model and immediately sent an alarm message to the user, such as "An abnormal fight has occurred in a certain area, please deal with it in time."

Claims

1. An intelligent spatial solution based on agent and multimodal understanding. This method uses a large language model as the core intelligent agent and combines the multimodal large model to perform comprehensive analysis of images and texts. The large language model and the multimodal large model can be directly used as open source or fine-tuned for spatial scenes. The characteristics are: The method specifically comprises: 1) System wake-up: The user speaks a specific wake-up word to the system through the microphone. The system monitors and recognizes the wake-up word in real time and activates the service process of the smart space assistant. After wake-up, the system confirms the activation status through voice prompts or light indications, and prompts the user to enter instructions; 2) Command input and analysis: The user inputs the task command in natural language, and the system converts the voice command into text through the speech recognition module and inputs it into the large language model agent; 3) Camera scene acquisition: The Agent instructs the camera in the corresponding area to capture real-time scene images or call historical videos according to task requirements. During the image capture process, the system automatically matches the area name or number in the instruction with the coverage of the camera to ensure that the data source is accurate; 4) Functional module pre-definition and design: support multiple functional modules and sustainable expansion; 5) Multimodal understanding and task execution: The captured scene images and parsed text instructions are input into the multimodal large model. The multimodal large model integrates visual information and text information to complete the understanding of the scene and the reasoning and execution of the task; 6) Result processing and feedback: After the multimodal large model completes the task, the feedback content is finally converted into voice broadcast to the user through the speech synthesis module, and it can also be displayed in text form on the visual interface; 7) Task recording and optimization: The system will record the task process and results in the background database for subsequent analysis and performance optimization.

2. The intelligent space solution based on agent and multimodal understanding according to claim 1 is characterized in that The specific process inside the language model Agent in step 2) is as follows: 1) Semantic parsing: The large language model parses the instruction text to extract key intent, parameters, and task content; 2) Task allocation: Based on the analysis results, the Agent allocates the task to the corresponding functional module and selects the camera or device to be called according to the task type.

3. The intelligent space solution based on agent and multimodal understanding according to claim 1 is characterized in that The functional modules in step 4) are as follows: 1) Spatial scene understanding module: supports single-region or multi-region scene analysis, assigns tasks to multimodal large models through agents, and outputs scene description results or task conclusions; 2) Target Counting Module: Counts the number of targets by region, supports users to specify regions in instructions, and calculates results through multi-modal large model analysis; 3) Target distance estimation module: Calculates the distance between two targets through depth estimation algorithm combined with multimodal large model understanding and positioning, which is suitable for measurement tasks within the coverage area of ​​a single camera; 4) Behavior understanding module: Analyze the behavior of designated personnel in a single area or multiple areas, such as identifying the behavior type and generating the corresponding behavior description; 5) Equipment control module: directly controls the on / off status of the equipment based on user instructions without the need for area designation; 6) Abnormal warning module: Actively or passively detect abnormal situations in the entire area. Passive detection is carried out by a background process that monitors the entire area once every 10 seconds. The camera monitors the entire area. The multimodal large model understands possible abnormal behaviors in the scene and issues alarms, such as fights between people, abnormal fireworks, etc., to achieve safety management through real-time monitoring and alarm notifications.

Images