Home assistant system based on large language model agent

Abstract

According to the home assistant system based on the large language model agent, a voice interaction module receives voice input of a user and converts the voice input into a text, multi-language recognition and voice synthesis are achieved through natural language processing, and natural and smooth two-way voice interaction is supported. The WeChat interaction module receives and processes text or voice information sent by a user through WeChat, and remote instruction input and information feedback are achieved. The core of the system is a plurality of functional Agent modules. The task analysis Agent analyzes semantics input by a user based on a large language model and dispatches a corresponding functional Agent to execute tasks, and the functions of information query, smart home control, information transmission, mobile navigation, monitoring and the like are covered. Personalized learning and optimization can be carried out according to a user-defined information base, use habits and historical records, the task execution precision and efficiency are improved, and the method is suitable for multi-scene application in the field of smart home.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a home assistant system based on a large language model intelligent agent. Background Art

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, smart home systems are increasingly used in modern homes. These systems integrate a variety of sensors and control devices to achieve multiple functions such as smart lighting, smart security, and temperature control, greatly improving the comfort and convenience of family life. However, most smart home systems on the market currently still have much room for improvement in terms of user experience and intelligence level, which is specifically reflected in the following aspects:

[0003] One of the main problems with existing smart home systems is the single interaction method. Many systems rely mainly on mobile applications or simple voice commands for operation, lacking natural and diverse interaction methods. Users need to constantly switch applications or devices during use, which is cumbersome and has a poor experience, especially for special groups such as the elderly and children.

[0004] Low functional integration is also an important problem facing the current smart home system. Smart home devices of different brands and types often work independently, lacking a unified system to coordinate and integrate the functions of these devices. Users need to operate multiple applications or devices separately, which is cumbersome and inefficient, and cannot achieve seamless linkage between devices.

[0005] Most smart home systems also have limited intelligence. Many smart home systems can only handle basic automation tasks and have limited understanding of complex user needs and natural language. The system cannot fully understand the user's intentions and contextual information, which leads to errors or failure to complete tasks when executing complex instructions. In addition, existing systems lack self-learning and optimization capabilities, and cannot make personalized adjustments based on user usage habits and historical records, and their intelligence level needs to be improved.

[0006] Therefore, a new solution for home assistant based on large language model agent is needed. Summary of the invention

[0007] In view of this, the embodiments of this specification provide a home assistant system based on a large language model agent. It aims to solve the shortcomings of existing smart home systems in terms of interaction mode, functional integration and intelligence level. The system comprehensively utilizes natural language processing, speech recognition, Internet of Things and robotics technology to provide a diversified, highly integrated and intelligent home assistant solution.

[0008] The embodiments of this specification provide the following technical solutions:

[0009] The embodiment of this specification provides a home assistant system based on a large language model agent, including a voice interaction module, a WeChat interaction module and an Agent module, wherein:

[0010] The voice interaction module is used to receive the user's voice input, convert the voice into text, and output voice feedback;

[0011] The WeChat interaction module is used to receive text or voice messages sent by users through WeChat, convert the received voice messages into text, and send system-generated text feedback through WeChat;

[0012] The Agent module includes a task analysis Agent and a plurality of functional Agents. The task analysis Agent is used to determine the functional Agent to be called according to the user's input information.

[0013] Furthermore, the WeChat interaction module needs to prepare a dedicated WeChat account for the family assistant to communicate with family members, mainly including:

[0014] The voice-to-text unit is used to convert the voice messages sent by users through WeChat into text.

[0015] The information sending unit is responsible for sending the text feedback generated by the system to the user via WeChat.

[0016] Through the WeChat interaction module, users can input commands and receive information through WeChat when it is inconvenient to use voice interaction. This module supports the sending and receiving of text messages and voice messages, greatly improving the flexibility and convenience of the system.

[0017] Furthermore, the Agent module is the core part of the system of the present invention, including a task analysis Agent and multiple functional Agents. The task analysis Agent is responsible for determining the functional Agent to be called according to the user's input information, and each functional Agent performs a specific task to meet the various needs of the user.

[0018] Among them, the task analysis agent determines the functional agent that needs to be called by analyzing the user's input information. The agent uses the natural language processing capabilities of the large language model to deeply understand the user's intentions and context information to ensure that the corresponding functional agent is called accurately. The task analysis agent is also responsible for coordinating the work between the functional agents to ensure the efficient operation of the entire system.

[0019] Furthermore, the online search agent is mainly used to process the user's query request. By calling the online search tool, the agent can obtain the required information and generate search results in text format. The online search agent can handle various types of queries, including news search, encyclopedia query and real-time hot search. Users can input query requests through voice or WeChat, and the system will return the corresponding search results according to the request.

[0020] Furthermore, the smart home agent is responsible for controlling and managing various smart devices in the home. The agent analyzes the user's task instructions, determines the user's intention, and performs corresponding operations, such as controlling home appliances, adjusting lighting and temperature control, etc. When the user's instructions are unclear, the smart home agent will make further inquiries to ensure that the correct operation is performed.

[0021] Furthermore, the communication agent is used to transmit information between the user and other family members. By parsing the user's task instructions, the communication agent can identify the sender and content of the information, and send the information to the designated object through the WeChat interaction module.

[0022] Furthermore, the mobile navigation agent determines the navigation point of the mobile according to the user's task instructions and navigates autonomously in the home environment. The agent combines personalized home mapping and marked points to accurately move in a known environment. The mobile navigation agent has an obstacle avoidance function and can safely avoid obstacles in the home to ensure the successful completion of the task.

[0023] Furthermore, the mobile monitoring agent is responsible for real-time monitoring of the home environment. According to the user's task instructions, the mobile monitoring agent can determine the navigation point and rotation angle of the mobile device to perform the monitoring task. The user can use WeChat instructions to let the system go to a certain map mark point and turn on the camera. The mobile monitoring agent will automatically go there and adjust the viewing angle to provide the required monitoring image. The user can also adjust the viewing angle of the monitoring device through subsequent instructions to ensure that the required monitoring information is obtained.

[0024] Furthermore, the home assistant system based on the large language model agent proposed in the present invention can improve the accuracy and efficiency of user interaction experience and task execution through personalized learning and optimization. The system can perform deep learning and intelligent optimization based on the user's customized information library, usage habits and historical records, thereby realizing personalized services and further improving user satisfaction and the intelligence level of the system.

[0025] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0026] The diverse interaction methods have greatly improved the user experience. The system supports voice and WeChat interaction, and users can choose the most convenient operation method according to actual conditions. The voice interaction module supports custom trigger words and multi-language processing. Users can interact with the system through simple voice commands to meet the needs of different family members, especially for special groups such as the elderly and children. It provides a more friendly user experience. The WeChat interaction module realizes information transmission between family members through a dedicated WeChat account. Even when it is inconvenient to use voice, users can input commands and receive information through WeChat, which increases the flexibility and convenience of the system.

[0027] The highly integrated design significantly improves the operating efficiency of the system. Through the collaborative work of the task parsing agent and multiple functional agents, the system realizes the unified management and control of various smart devices in the home. Users do not need to switch multiple applications, but can complete information query, smart home control, information transmission, mobile navigation and monitoring and other functions through one platform. This highly integrated design not only simplifies the operation process, but also improves the collaborative work efficiency between devices, and the user experience is smoother and more efficient.

[0028] With the powerful natural language processing capabilities of the large language model, the system has the advantages of intelligent processing and personalized services. The task parsing agent can deeply understand the user's intentions and context information, ensure the accurate calling of the corresponding functional agent, and realize the parsing and processing of complex instructions. The functional agent can provide precise services according to the user's needs: the network search agent can quickly obtain the required information, the smart home agent can accurately perform home appliance control tasks, the communication agent agent can efficiently transmit information, and the mobile navigation agent and mobile monitoring agent can independently complete navigation and monitoring tasks. The system also has self-learning and optimization capabilities, and continuously improves services based on user usage habits and historical records to provide a more personalized and intimate experience.

[0029] The present invention provides a diversified, highly integrated and intelligent home assistant system, which significantly improves the user experience and operation efficiency of the smart home system and meets the diverse needs of modern families for smart life. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1It is a flowchart of a home assistant system based on a large language model agent according to an embodiment of the present application;

[0032] Figure 2 This is a structural diagram of a voice interaction module in a home assistant system based on a large language model agent in an embodiment of the present application;

[0033] Figure 3 It is a structural diagram of a WeChat interaction module in a home assistant system based on a large language model intelligent agent in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0035] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0036] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0037] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0038] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the examples can be practiced without these specific details.

[0039] Most smart home systems on the market still have much room for improvement in terms of user experience and intelligence level, which is specifically reflected in the following aspects:

[0040] One of the main problems with existing smart home systems is the single interaction method. Many systems rely mainly on mobile applications or simple voice commands for operation, lacking natural and diverse interaction methods. Users need to constantly switch applications or devices during use, which is cumbersome and has a poor experience, especially for special groups such as the elderly and children.

[0041] Low functional integration is also an important problem facing the current smart home system. Smart home devices of different brands and types often work independently, lacking a unified system to coordinate and integrate the functions of these devices. Users need to operate multiple applications or devices separately, which is cumbersome and inefficient, and cannot achieve seamless linkage between devices.

[0042] Most smart home systems also have limited intelligence. Many smart home systems can only handle basic automation tasks and have limited understanding of complex user needs and natural language. The system cannot fully understand the user's intentions and contextual information, which leads to errors or failure to complete tasks when executing complex instructions. In addition, existing systems lack self-learning and optimization capabilities, and cannot make personalized adjustments based on user usage habits and historical records, and their intelligence level needs to be improved.

[0043] In view of this, the inventors found that the large language model has powerful natural language processing capabilities, can understand and generate natural language text, and provides a technical basis for the intelligent improvement of smart home systems. By introducing the large language model, the system's ability to understand and process user instructions can be significantly enhanced, achieving more natural and efficient user interaction. The emergence of the large language model has brought new development opportunities for smart home systems.

[0044] Based on this, the embodiments of this specification propose a home assistant system based on a large language model agent, aiming to solve the shortcomings of existing smart home systems in terms of interaction mode, functional integration and intelligence level. The system comprehensively utilizes natural language processing, speech recognition, Internet of Things and robotics technology to provide a diversified, highly integrated and intelligent home assistant solution.

[0045] The technical solutions provided by various embodiments of the present application are described below in conjunction with the accompanying drawings.

[0046] The present invention proposes a home assistant system based on a large language model agent. The specific implementation process is shown in the following embodiments:

[0047] Reference Figure 1 As shown, the method proposed in the embodiment of the present invention includes:

[0048] 1. Obtain task information issued by users, mainly in two ways: voice commands and WeChat messages;

[0049] 2. Analyze the task, determine the user's intention, and select the appropriate functional agent. Based on the obtained task information, determine the user's intention and then select the appropriate functional agent.

[0050] 3. Use appropriate functional agents to serve users, mainly including: Internet search agent, smart home agent, communication agent, mobile navigation agent and mobile monitoring agent.

[0051] Furthermore, in the embodiment of the present invention, a voice interaction module and a WeChat interaction module are used in the process of obtaining task information issued by a user.

[0052] The main function of the voice interaction module is to receive the user's voice input, convert it into text, and convert the system's feedback into voice output to achieve two-way voice interaction. Figure 2 As shown, the voice interaction module includes a voice recognition unit and a voice synthesis unit, supports multi-language processing, and has the function of customizing trigger words.

[0053] The voice recognition unit captures the user's voice commands through a microphone and uses advanced voice recognition algorithms to convert the voice into text. The specific steps are as follows:

[0054] Audio capture: The microphone captures the user's voice input in real time and converts it into digital signals;

[0055] Preprocessing: Preprocess the captured audio signals, including noise reduction, filtering, and segmentation, to improve recognition accuracy;

[0056] Feature extraction: Use speech signal processing techniques (such as MFCC) to extract features from the preprocessed audio signal for subsequent recognition process;

[0057] Model matching: Using the trained speech recognition model, the extracted features are matched with the language patterns in the model and converted into text output.

[0058] The speech synthesis unit is responsible for converting the text feedback generated by the system into speech and playing it to the user through the speaker. The specific steps are as follows:

[0059] Text processing: Process the input text, including word segmentation, stress marking and intonation, to generate speech text suitable for synthesis;

[0060] Speech generation: Use deep learning-based speech synthesis models (such as Tacotron and WaveNet) to convert processed text into natural and fluent speech signals;

[0061] Audio output: The generated voice signal is played to the user through the speaker to achieve voice feedback.

[0062] The voice interaction module supports multi-language processing and can recognize and process voice input in multiple languages.

[0063] The specific implementation includes:

[0064] Multilingual model: It uses a multilingual pre-trained model that can recognize speech input in different languages ​​and process it accordingly;

[0065] Language recognition: The system can automatically detect the language type of the input speech and call the corresponding language model for processing.

[0066] In addition, the voice interaction module has the function of customizing trigger words. Users can start the system with preset trigger words, putting it into standby mode and waiting for further voice commands. The specific implementation includes:

[0067] Trigger word detection: The user's voice is monitored in real time through a keyword detection algorithm. When a preset trigger word is detected, the system is activated.

[0068] State switching: After the system enters the standby state, it continues to listen to the user's subsequent commands and performs corresponding processing.

[0069] The voice interaction module realizes natural language interaction between users and the system through the collaborative work of the voice recognition unit and the voice synthesis unit. The multi-language processing and custom trigger word functions further enhance the flexibility and user experience of the system, enabling it to adapt to the needs of different users and provide intelligent, convenient and personalized services for families.

[0070] Furthermore, the WeChat interaction module aims to realize information transmission between users and the system through the WeChat platform, and provide two-way interaction of text and voice. The main functions of the module include receiving instructions sent by users through WeChat, converting them into text that can be processed by the system, and sending feedback generated by the system to users through WeChat. Figure 3 As shown, the WeChat interaction module mainly includes a voice processing unit and an information sending unit:

[0071] The voice-to-text unit is responsible for converting the voice messages sent by users through WeChat into text. The specific implementation steps are as follows:

[0072] Voice reception: The WeChat platform receives the voice message sent by the user and transmits it to the system background;

[0073] Audio decoding: decode the received voice information and extract audio data;

[0074] Speech recognition: Use pre-trained speech recognition models to convert audio data into text. This process involves steps such as speech signal processing, feature extraction, and model matching to ensure the accuracy and real-time nature of the recognition results;

[0075] Text output: The recognized text is passed to the system for further processing.

[0076] The information sending unit is responsible for sending the text feedback generated by the system to the user through WeChat. The specific implementation steps are as follows:

[0077] Message formatting: Format the generated text information to meet the message sending requirements of the WeChat platform;

[0078] Message sending: Formatted text messages are sent to users through the API interface of the WeChat platform to ensure that users can receive feedback from the system in a timely manner.

[0079] The WeChat interaction module greatly improves the flexibility and convenience of the system. When it is inconvenient for users to use voice interaction, they can use WeChat to input commands and receive information. This module supports the sending and receiving of text messages and voice messages, adapting to user needs in different scenarios.

[0080] Furthermore, in the embodiment of the present invention, parsing tasks, judging user intentions and selecting appropriate functional agents mainly rely on the task parsing agent.

[0081] The task parsing agent is responsible for understanding the user's input and determining the appropriate functional agent to perform the task. The task parsing agent mainly uses natural language processing technology to accurately identify the user's intention and efficiently allocate tasks. The specific implementation plan is as follows:

[0082] 1) Task Analysis The Agent first receives the text input after being processed by the voice interaction module or WeChat interaction module, and then preprocesses it. The preprocessing steps include word segmentation, removal of stop words, and part-of-speech tagging for subsequent semantic analysis.

[0083] 2) Task parsing Agent uses a large language model to understand natural language. The main steps are as follows:

[0084] By performing semantic analysis on the preprocessed text, key entities, relations, and intents are extracted. The large language model can understand complex language structures and contextual information, thereby accurately identifying the user's intent;

[0085] Use deep learning models to classify the intent of user input. The intent classification model is trained based on a large amount of labeled data and can identify various common home assistant tasks, such as information query, device control, information transmission, etc.

[0086] The user's historical conversations and current context information are combined to further refine and confirm the user's intentions. The context understanding ability of the large language model enables the system to handle continuous conversations and complex instructions.

[0087] 3) The task analysis agent determines the functional agent that needs to be called based on the identified user intention and assigns tasks. The specific steps are as follows:

[0088] According to the user's intention, the corresponding functional agent is matched. For example, for information query intention, the network search agent is called; for smart home control intention, the smart home agent is called;

[0089] Pass the parsed task parameters (such as query keywords, device names, operation instructions, etc.) to the corresponding functional Agent to ensure the accuracy of task execution;

[0090] In the case of multiple functional agents working together, the task resolution agent is responsible for coordinating the work between the agents. For example, when information query and user communication need to be performed simultaneously, the network search agent and the communication agent are called separately, and their task execution order and result feedback are coordinated.

[0091] The task parsing agent achieves accurate parsing of user input and efficient task allocation through natural language understanding, intent classification and task allocation. Its core lies in utilizing the powerful natural language processing capabilities of the large language model to ensure that the system can intelligently and accurately understand and execute various user needs, thereby providing personalized and convenient home services.

[0092] Among them, in order to meet user needs, it is necessary to use appropriate functional agents to serve users, mainly including: network search agent, smart home agent, communication agent, mobile navigation agent and mobile monitoring agent.

[0093] Furthermore, the online search agent is responsible for processing the user's query request, obtaining the required information through the Internet, and generating readable search results to feed back to the user. The specific implementation plan is as follows:

[0094] 1) Query request reception and preprocessing

[0095] The online search agent first receives the query request from the task analysis agent. The request contains the user's query keywords and related context information. The preprocessing steps include:

[0096] Keyword extraction: Extract the main keywords and phrases from the query request to ensure that the core content of the query is clear;

[0097] Contextual analysis: Utilize the contextual understanding capabilities of large language models, combine the user's historical conversations and current situation, and refine and clarify query requirements.

[0098] 2) Query execution

[0099] The online search agent executes queries by calling external search engine APIs (such as Google, Bing, etc.) or specific information sources (such as Wikipedia, news websites, etc.). The specific steps are as follows:

[0100] API call: construct a query request based on the preprocessed keywords and call the search engine API to obtain search results;

[0101] Information source selection: Select the most appropriate information source based on the query type and user needs. For example, encyclopedia queries call the Wikipedia API, and news searches call the news website API.

[0102] 3) Search result processing

[0103] After obtaining the original search results, the online search agent screens and processes the results to ensure the accuracy and readability of the information. The specific steps include:

[0104] Result screening: Perform a preliminary screening of search results to remove irrelevant or low-quality information. Use the understanding ability of the large language model to perform semantic analysis on the results and retain the most relevant content;

[0105] Summary generation: Generate a concise summary of the filtered information, extract key information, and ensure that users can quickly obtain the required content;

[0106] Formatting: Format the summary and related information to generate easy-to-read text feedback, which is convenient for the system to further output to the user.

[0107] 4) User feedback generation and delivery

[0108] The search results generated by the online search agent need to be delivered to the user. The specific steps are as follows:

[0109] Feedback integration: Integrate the processed search results into a coherent text, including query keywords, abstracts, and related links;

[0110] Natural language generation: Use a large language model to generate natural and fluent feedback text to ensure a good user experience;

[0111] Output transfer: Through the task analysis agent, the final generated feedback is passed to the voice interaction module or WeChat interaction module, and fed back to the user in the form of voice or text.

[0112] The networked search agent achieves a rapid response to user information needs through efficient query execution and sophisticated result processing. It uses the powerful natural language processing capabilities of the large language model to ensure the accuracy and readability of search results, provide intelligent and convenient information query services, and greatly improve the practicality and user experience of the home assistant system.

[0113] Furthermore, the smart home agent is responsible for controlling and managing various smart devices in the home, and through natural language processing technology, it realizes convenient control and management of smart homes. The specific implementation plan is as follows:

[0114] 1) Command reception and preprocessing

[0115] The smart home agent first receives control instructions from the task parsing agent, which are input by the user through voice or text. The preprocessing steps include:

[0116] Command parsing: Use a large language model to parse the commands and extract the controlled device name, operation type, and related parameters (such as brightness, temperature, etc.).

[0117] Contextual understanding: Combine the user's historical instructions with the current context to further clarify the user's intention. For example, the user may say "dim the lights in the living room", and the system needs to understand that "dim" means to reduce the brightness, and "the lights in the living room" is a specific device.

[0118] 2) Device identification and status query

[0119] The smart home agent needs to identify the devices involved in the user's instructions and query their current status. The specific steps are as follows:

[0120] Device matching: Match the devices mentioned in the command in the smart home device list to determine the specific control target. The system should support multiple smart home protocols (such as Zigbee, Z-Wave, Wi-Fi, etc.) to ensure device compatibility;

[0121] Status query: Call the device's status query interface to obtain the current device status information (such as switch status, brightness, temperature, etc.) in order to perform reasonable control operations.

[0122] 3) Control instruction execution

[0123] According to the parsed instructions and the current status of the device, the smart home agent performs specific control operations. The specific steps include:

[0124] Command generation: Generate corresponding device control commands, such as turning lights on and off, adjusting temperature, setting timers, etc. Command generation needs to consider the specific operation protocol and parameter format of the device;

[0125] Command sending: The generated control commands are sent to the corresponding devices through the smart home control protocol. The system needs to ensure the reliability and timeliness of command sending to ensure user experience;

[0126] Result feedback: Get feedback information after the device executes the instruction, confirm whether the operation is successful, and pass the result to the user.

[0127] 4) User feedback generation and delivery

[0128] The smart home agent generates feedback information based on the operation results of the device and transmits it to the user. The specific steps are as follows:

[0129] Result integration: Integrate the execution results of the device into a coherent text, including information such as device name, operation type and execution result;

[0130] Natural language generation: Use a large language model to generate natural and fluent feedback text to ensure that users can clearly understand the operation results;

[0131] Output transfer: Through the task analysis agent, the final generated feedback is passed to the voice interaction module or WeChat interaction module, and fed back to the user in the form of voice or text.

[0132] Smart home agent enables users to conveniently control smart home devices through efficient command parsing, device identification, control execution and result feedback. By using the natural language processing capabilities of the large language model, the system can accurately understand the user's intentions and perform corresponding operations, providing intelligent and personalized home services, significantly improving the convenience and comfort of family life.

[0133] Furthermore, the communication agent is responsible for transmitting information between the user and other family members, and realizes efficient information transmission and communication management through natural language processing technology. The specific implementation plan is as follows:

[0134] 1) Command reception and analysis

[0135] The communication agent first receives the task instructions from the task analysis agent. These instructions are input by the user through voice or text, and the content involves the object and specific information of the information transmission. The specific steps include:

[0136] Command reception: obtaining the user's command information, including the sender (such as the name or identity of a family member) and the content of the message;

[0137] Intent parsing: Use a large language model to semantically parse instructions and extract key elements, such as the object of the message and the content of the message. Contextual understanding ensures that the system can correctly parse complex instructions, such as "Send a message to mom saying that the meal is ready and go home slowly";

[0138] 2) Information generation and formatting

[0139] According to the parsed instructions, the communication agent generates the information to be transmitted. The specific steps are as follows:

[0140] Information confirmation: confirm the accuracy and completeness of information content to ensure that the user's intention is fully expressed;

[0141] Content formatting: Format the information content to adapt to the requirements of the WeChat platform.

[0142] 3) Information sending and transmission

[0143] The communication agent sends the information to the designated recipient. The specific steps include:

[0144] Message sending: Call the corresponding API interface to send the formatted information to the recipient. The system needs to ensure the reliability and timeliness of message sending and handle possible sending failures (such as network problems);

[0145] Send confirmation: Get the sending result feedback, confirm whether the message is successfully delivered, and record the sending status.

[0146] 4) Result feedback and user notification

[0147] The communication agent feeds back the message sending result to the user. The specific steps are as follows:

[0148] Result integration: integrate the message sending status information into a concise text, including the sending time, recipient and sending result, etc.;

[0149] Natural language generation: Use a large language model to generate natural and fluent feedback text to ensure that users can clearly understand the sending status;

[0150] Feedback transmission: Through the task analysis agent, the generated feedback information is transmitted to the voice interaction module or WeChat interaction module, and the user is notified in the form of voice or text.

[0151] The communication agent achieves efficient information transmission between users and family members through efficient command parsing, information generation, transmission and result feedback. Using the natural language processing capabilities of the large language model, the system can accurately understand the user's intentions and perform corresponding operations, providing intelligent and convenient information communication services, significantly improving the communication efficiency and user experience between family members.

[0152] The mobile navigation agent is responsible for autonomous navigation in the home environment, and achieves precise and safe movement and task execution through natural language processing and advanced navigation technology. The specific implementation plan is as follows:

[0153] 1) Navigation instruction reception and analysis

[0154] The mobile navigation agent first receives navigation instructions from the task parsing agent, which contain the user's task requirements and navigation goals. The specific steps include:

[0155] Command reception: Obtain the user's navigation command information, including the target location (such as kitchen, living room), task type (such as item delivery), etc.

[0156] Intent parsing: Use a large language model to parse instructions and extract key elements, such as the target location, navigation path, and specific tasks. Contextual understanding capabilities ensure that the system can correctly parse complex instructions, such as "Go to the kitchen and take two cups to the living room."

[0157] 2) Home environment modeling and positioning

[0158] In order to achieve accurate navigation, the mobile navigation agent needs to model and locate the home environment. The specific steps are as follows:

[0159] Environmental modeling: Use sensors (such as lidar, cameras) to scan the home environment and generate a detailed floor plan and 3D model. The model includes information such as room layout, furniture location, and dynamic obstacles;

[0160] Positioning technology: Using SLAM (Simultaneous Localization and Mapping) technology, the robot can update its position in the environment in real time. Combined with data from sensors such as odometers and IMUs (Inertial Measurement Units), it can achieve high-precision positioning and navigation.

[0161] 3) Path planning and obstacle avoidance

[0162] The mobile navigation agent plans the optimal path based on the user's instructions and the environment model, and avoids obstacles during movement. The specific steps include:

[0163] Path planning: Based on path planning algorithms such as Dijkstra algorithm and A* algorithm, calculate the optimal path from the current location to the target location, taking into account factors such as path length, time and safety;

[0164] Real-time obstacle avoidance: Use lidar, ultrasonic sensors, etc. to detect the surrounding environment in real time, identify dynamic obstacles (such as pets and children), and adjust the navigation path to ensure safe movement.

[0165] 4) Navigation execution and task completion

[0166] The mobile navigation agent performs specific navigation tasks to ensure the successful completion of the tasks. The specific steps include:

[0167] Task execution: Move along the planned path, adjust speed and direction in real time, avoid obstacles, and reach the target location. Maintain a stable navigation state during execution to avoid collisions and falls;

[0168] Mission completion: After arriving at the target location, complete specific tasks according to user instructions, such as item delivery, etc. Use robotic arms, grippers, etc. to perform tasks to ensure that tasks are completed accurately and efficiently.

[0169] 5) User feedback generation and delivery

[0170] The mobile navigation agent feeds back the navigation and task execution results to the user. The specific steps are as follows:

[0171] Result integration: integrate the navigation process and task execution status information into a concise text, including movement path, arrival time and task completion status;

[0172] Natural language generation: Use a large language model to generate natural and fluent feedback text to ensure that users can clearly understand navigation and task execution results;

[0173] Feedback transmission: Through the task analysis agent, the generated feedback information is transmitted to the voice interaction module or WeChat interaction module, and the user is notified in the form of voice or text.

[0174] Furthermore, the mobile navigation agent achieves autonomous navigation and task completion in the home environment through efficient command parsing, environment modeling, path planning and task execution. Using the natural language processing capabilities of the large language model and advanced navigation technology, the system can accurately understand the user's intentions and perform corresponding operations, providing intelligent and convenient navigation services, significantly improving the safety and convenience of family life.

[0175] The mobile monitoring agent is responsible for real-time monitoring of the home environment, and achieves flexible and accurate monitoring tasks through autonomous navigation and smart cameras. The specific implementation plan is as follows:

[0176] 1) Monitoring command reception and analysis

[0177] The mobile monitoring agent first receives monitoring instructions from the task parsing agent, which contain the user's monitoring requirements and target location. The specific steps include:

[0178] Command reception: Obtain the user's monitoring command information, including the target location (such as living room, door), monitoring task type (such as scheduled monitoring, mobile patrol), etc.

[0179] Intent parsing: Use a large language model to parse the command and extract key elements, such as the target location, monitoring time, and specific tasks. Contextual understanding ensures that the system can correctly parse complex commands, such as "check the windows in the living room."

[0180] 2) Home environment modeling and positioning

[0181] In order to achieve accurate monitoring, the mobile monitoring agent needs to model and locate the home environment. The specific steps are as follows:

[0182] Environmental modeling: Use sensors (such as lidar, cameras) to scan the home environment and generate a detailed floor plan and 3D model. The model includes information such as room layout, furniture location, and dynamic obstacles;

[0183] Positioning technology: Using SLAM (Simultaneous Localization and Mapping) technology, it updates its position in the environment in real time, and combines data from sensors such as odometers and IMUs (Inertial Measurement Units) to achieve high-precision positioning and navigation.

[0184] 3) Path planning and obstacle avoidance

[0185] The mobile monitoring agent plans the optimal path to reach the monitoring location based on the user's instructions and the environment model, and avoids obstacles during the movement. The specific steps include:

[0186] Path planning: Based on path planning algorithms such as Dijkstra algorithm and A* algorithm, calculate the optimal path from the current location to the target location, taking into account factors such as path length, time and safety;

[0187] Real-time obstacle avoidance: Use lidar, ultrasonic sensors, etc. to detect the surrounding environment in real time, identify dynamic obstacles (such as pets and children), and adjust the navigation path to ensure safe movement.

[0188] 4) Monitor task execution

[0189] After the mobile monitoring agent arrives at the target location, it performs specific monitoring tasks to ensure monitoring results and task completion. The specific steps include:

[0190] Camera adjustment: After reaching the target location, the camera angle and focal length are automatically adjusted to ensure coverage of the predetermined monitoring area. The camera can be rotated horizontally and vertically to provide all-round monitoring;

[0191] Real-time monitoring: Real-time monitoring is carried out through cameras, and the monitoring images are transmitted to the user's terminal device (such as mobile phones, computers). Users can view the real-time monitoring images through WeChat or dedicated applications;

[0192] Anomaly detection: Use image recognition technology to analyze surveillance images in real time, detect abnormal situations (such as strangers breaking in, fire, etc.), and notify users immediately.

[0193] 5) User feedback generation and delivery

[0194] The mobile monitoring agent feeds back the execution results of the monitoring task and the real-time monitoring screen to the user. The specific steps are as follows:

[0195] Result integration: Integrate the monitoring process and task execution status information into a concise text, including movement path, arrival time and monitoring screen, etc.;

[0196] Natural language generation: Use a large language model to generate natural and fluent feedback text to ensure that users can clearly understand monitoring and task execution results;

[0197] Feedback transmission: Through the task analysis agent, the generated feedback information is transmitted to the voice interaction module or WeChat interaction module, and the user is notified in the form of voice or text.

[0198] The mobile monitoring agent achieves flexible monitoring and real-time feedback in the home environment through efficient command parsing, environment modeling, path planning and task execution. Using the natural language processing capabilities of the large language model and advanced monitoring technology, the system can accurately understand the user's intentions and perform corresponding operations, providing intelligent and convenient monitoring services, significantly improving the safety and security of family life.

[0199] In addition, the home assistant system based on the large language model agent proposed in the present invention can improve the accuracy and efficiency of user interaction experience and task execution through personalized learning and optimization. The system can perform deep learning and intelligent optimization based on the user's customized information library, usage habits and historical records, thereby realizing personalized services and further improving user satisfaction and the intelligence level of the system. The specific implementation plan is as follows:

[0200] 1) Establishment and management of personalized information database

[0201] The system first establishes a personalized information database for users. Users can manually input or automatically collect information such as personal preferences, commonly used devices, daily routines, etc. The system analyzes this information to form a personalized profile for users.

[0202] 2) Usage habits and history analysis

[0203] The system can use sensors and log recording functions to continuously monitor users' operating behaviors and usage frequency. At the same time, it can use big data analysis technology to classify and identify patterns of users' usage habits and historical records to form a user behavior model for personalized optimization.

[0204] 3) Personalized learning and optimization

[0205] The system uses a large language model and deep learning algorithms to analyze user input and feedback, identify changes in user preferences and needs, and adjust the system's interaction methods and task execution strategies based on the learning results, such as recommending the most commonly used device control options or providing personalized information push.

[0206] The present invention provides a family assistant system based on a large language model agent, which realizes the intelligence, convenience and personalization of family life by integrating natural language processing, speech recognition, Internet of Things and robotics technology. The system consists of a voice interaction module, a WeChat interaction module and an Agent module, which can meet the diverse needs of users.

[0207] The voice interaction module uses advanced speech recognition and synthesis technology to achieve two-way natural language interaction, support multi-language processing and custom trigger words, and improve the convenience and flexibility of user experience. The WeChat interaction module provides the sending and receiving functions of text and voice messages through the WeChat platform, providing users with an alternative solution when voice interaction is inconvenient, enhancing the applicability and practicality of the system.

[0208] The Agent module is the core of the system, which includes the task parsing agent and multiple functional agents, such as the network search agent, smart home agent, communication agent, mobile navigation agent and mobile monitoring agent. The task parsing agent uses the powerful natural language processing capabilities of the large language model to accurately understand the user's intentions, coordinate the call of functional agents, and ensure the efficient operation of system functions. Each functional agent is responsible for tasks such as information query, smart home control, information transmission, mobile navigation and monitoring, which comprehensively improves the service capabilities of the home assistant. At the same time, with the further development of technology and changes in user needs, the system can also be updated and improved to adapt to a wider range of home application scenarios and personalized needs.

[0209] In summary, the present invention realizes the efficiency, convenience and personalized service of the home assistant system through intelligent technology and modular design, and demonstrates broad application prospects and development potential in the field of smart home.

[0210] The same or similar parts between the various embodiments in this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the product embodiment described later, since it corresponds to the method, the description is relatively simple, and the relevant parts can be referred to the partial description of the system embodiment.

[0211] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A home assistant system based on a large language model agent, characterized in that: It includes voice interaction module, WeChat interaction module and Agent module, among which: The voice interaction module is used to receive the user's voice input, convert the voice into text, and output voice feedback; The WeChat interaction module is used to receive text or voice messages sent by users through WeChat, convert the received voice messages into text, and send system-generated text feedback through WeChat; The Agent module includes a task analysis Agent and a plurality of functional Agents. The task analysis Agent is used to determine the functional Agent to be called according to the user's input information.

2. The home assistant system based on a large language model agent according to claim 1, characterized in that: The voice interaction module includes a custom trigger word for starting the voice recognition function of the home assistant and supports multi-language processing to recognize and process voice input in multiple languages.

3. The home assistant system based on a large language model agent according to claim 1, characterized in that: The WeChat interaction module prepares a WeChat account for the family assistant, which is used to transmit information to family members, supports the reception of text messages and voice messages, and converts the received voice messages into text.

4. The home assistant system based on a large language model agent according to claim 1, characterized in that: The functional agents in the Agent module include network search agent, smart home agent, communication agent, mobile navigation agent and mobile monitoring agent.

5. The home assistant system based on a large language model agent according to claim 4, characterized in that: The online search agent obtains information by calling an online search tool, generates search results in text format, converts and sends the information according to the format of the information sent by the user, and includes news search, encyclopedia query, and real-time hot search.

6. The home assistant system based on a large language model agent according to claim 4, characterized in that: The smart home agent analyzes the user's task instructions and makes further inquiries when the user's intention is unclear until the user's intention is determined through interaction, and then performs the corresponding smart home control tasks, including controlling home appliances, adjusting lighting, and temperature control.

7. The home assistant system based on a large language model agent according to claim 4, characterized in that: The communication agent Agent parses the user's task instructions, identifies the sender of the information and the content to be sent, and sends the information through the WeChat interaction module.

8. The home assistant system based on a large language model agent according to claim 4, characterized in that: The mobile navigation agent determines the navigation point of the mobile according to the user's task instruction, combines the personalized home map and the marked points, and performs the mobile task to autonomously navigate in the home environment.

9. The home assistant system based on a large language model agent according to claim 4, characterized in that: The mobile monitoring agent determines the moving navigation point and the camera rotation angle according to the user's task instructions, performs the monitoring task, and adjusts the viewing angle according to subsequent instructions to monitor the home environment in real time.

10. The home assistant system based on a large language model agent according to claim 1, characterized in that: Based on the user's customized information base, usage habits, and historical records, personalized learning and optimization are performed to improve the accuracy and efficiency of the interactive experience and task execution.

Images