A multi-agent-based household appliance instruction understanding and operation instruction generation method

Abstract

This invention discloses a method for understanding home appliance instruction manuals and generating operation instructions based on multi-agent systems. The method includes: receiving user instructions and acquiring multimodal perception information of the target home appliance panel; determining target instruction manual knowledge based on the perception information and user instructions using a hierarchical matching strategy, and mapping it to interactive components to obtain structured instruction manual knowledge; inputting user instructions, structured instruction manual knowledge, perception information, and device status into a multi-agent system, and generating an operation plan sequence through a planning agent; using a reflective agent to perform feasibility and safety assessments, guiding execution and entering the next loop if successful, and triggering a clarification and exploration process to correct the plan if unsuccessful or uncertain, until a complete operation sequence is generated. This invention solves the problems of missing instruction manuals or difficulty in aligning controls through multi-agent collaboration and a perception-planning-reflection closed loop, enabling the output of executable textual operation guidance and lowering the user's operational threshold.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a method for understanding and generating operating instructions for home appliances based on multi-agent systems. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale pre-trained models have achieved remarkable results in natural language processing, image recognition, and many other fields. These models, through deep learning technology, extract knowledge from massive amounts of data and possess powerful understanding and generation capabilities. However, the application of single models in complex environments still has limitations.

[0003] To address this problem, multi-agent systems have emerged. Multi-agent systems involve multiple agents collaborating to complete tasks, enabling them to share information, allocate tasks, and make decisions within specific scenarios. This structure not only improves the system's flexibility and robustness but also allows agents to perform operations more effectively in dynamic and uncertain environments.

[0004] With the widespread adoption of smart home devices, appliances such as microwave ovens, air fryers, ovens, washing machines, coffee makers, and dishwashers are becoming increasingly feature-rich. Panel interaction has evolved from mechanical knobs and physical buttons to touchscreens, multi-level menus, and combination modes. Appliance operation heavily relies on instruction manuals: these manuals typically contain extensive text descriptions, illustrations, tables, parameter ranges, and precautions, and different brands and models show significant differences in control naming, icon symbols, menu hierarchy, upper / lower parameter limits, and safety warnings. Ordinary users often need to repeatedly consult the manual when faced with complex appliances, and even then, misunderstandings can lead to operational failures or trigger unsafe behaviors (such as starting without closing the door, heating at incorrect power, using the wrong washing program, or setting the wrong temperature / time). In real-world scenarios, instruction manuals may be missing, damaged, or incompatible with the current device model; even if a manual exists, inconsistencies may arise due to version differences, language differences, regional differences, or model updates. In existing technologies, some systems rely on preset scripts or fixed question-and-answer libraries, covering only a few models and limited functions. While some assistants based on large models can provide general explanations, they lack reliable identification of specific panel controls, stable alignment of controls with instruction manual information, and explicit modeling of parameter / state constraints. Therefore, they struggle to consistently output executable and verifiable operation steps. Especially when there is no instruction manual or only manuals for similar product categories, existing systems cannot guide users to complete precise settings on specific panels. Therefore, a technical solution is needed that can understand instruction manuals, construct category-level general instruction manuals, complete them online when necessary, and generate reliable operation sequences by combining panel images and user commands. Summary of the Invention

[0005] The main objective of this invention is to provide a method for understanding and generating operating instructions for home appliances based on multi-agent systems.

[0006] Another objective of this invention is to provide a device for understanding and generating operating instructions for home appliances based on multiple agents.

[0007] The third objective of this invention is to provide a computer device.

[0008] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention proposes a method for understanding and generating operating instructions for home appliances based on multi-agent systems, comprising:

[0010] After receiving a user instruction, the system acquires multimodal sensing information of the target home appliance panel, which includes the position, appearance features, and status readings of interactive components on the panel. Based on the multimodal perception information and user instructions, the target instruction manual knowledge is determined through a hierarchical matching strategy, and the functional items in the target instruction manual knowledge are mapped to the interactive components in the multimodal perception information to obtain structured instruction manual knowledge. The user instructions, the structured specification knowledge, the multimodal perception information, and the current device status are input into the multi-agent system, and the planning agent generates an operation plan sequence containing a description of the target control and operation actions; the multi-agent system includes at least a perception agent, a status acquisition agent, a planning agent, and a reflection agent; The feasibility and safety of the operation plan sequence are evaluated by the reflective agent. If the evaluation is passed but the task is not completed, the user is guided to execute and enter the next round of perception planning loop. If the evaluation is not passed or there is uncertainty in the state, the clarification exploration process is triggered to correct the operation plan sequence until a complete operation plan sequence is generated.

[0011] In one embodiment of the present invention, acquiring the multimodal sensing information of the target home appliance panel includes: Collect image data of the target home appliance panel, and use the target detection model to identify the bounding box coordinates and component category labels of all interactive components on the panel; Use text recognition to extract the label text and display screen readings in the vicinity of the component; The bounding box coordinates and component category labels are spatially associated and mapped with the identification text and display screen readings. The associated recognition results are then input into a multimodal large model for joint semantic reasoning of text and images to generate encapsulated perception information containing functional descriptions, appearance features, and indicator light status of each component, which serves as the multimodal perception information.

[0012] In one embodiment of the present invention, a general specification knowledge base is constructed before receiving user instructions, including: Collect multiple original instruction manuals for different models of home appliances and extract the names and descriptions of their functional components; The extracted functional component names are mapped to semantic embedding vectors using an embedding model, and clustering algorithms are used to group names with semantic similarity higher than a set threshold into the same functional component cluster. For each functional component cluster, the name with the highest average semantic similarity within the cluster is selected as the unified name. The descriptions of the function in each instruction manual are merged and conflicting information is removed to construct the general instruction manual knowledge base.

[0013] In one embodiment of the present invention, determining the target specification knowledge through a hierarchical matching strategy includes: The system queries the original instruction manual library and calculates the semantic similarity between the component names in the multimodal perception information and the instruction manual control fields in the library. If there is a model matching instruction manual with a similarity higher than a preset threshold, it is directly selected as the original instruction manual. If no matching instruction manual is available or the highest similarity is below a preset threshold, the general instruction manual knowledge base is invoked. The component name is converted into an embedding vector using a language model, and the similarity with the part name corresponding to each entry in the library is calculated. The general instruction manual entry with the highest similarity is selected as the basic instruction manual knowledge. If the basic instruction manual knowledge does not include the parameter range or specific controls involved in the user instructions, then search keywords are constructed to call external network resources for supplementary search, and the missing operation steps and safety constraint information are extracted from the search results and integrated to generate the final target instruction manual knowledge.

[0014] In one embodiment of the present invention, the step of mapping functional entries in the target specification knowledge to interactive components in multimodal perception information to obtain structured specification knowledge includes: Analyze the layout structure of the target instruction manual and extract the control names, icon features, operation steps, parameter ranges, and security constraints. Calculate the semantic similarity and visual feature matching degree between the extracted control names and icon features and the interactive components in the multimodal perception information, and establish the correspondence between the manual controls and the panel physical components. Based on the aforementioned correspondence, the functional items, operation steps, parameter ranges, and safety constraints in the instruction manual knowledge are associated with the corresponding interactive components to generate structured instruction manual knowledge containing preconditions and postconditions.

[0015] In one embodiment of the present invention, the step of generating an operation plan sequence containing a target control description and operation actions through a planning agent includes: The state acquisition agent, based on the structured specification knowledge and multimodal perception information, infers the observable and unobservable state variables of the current home appliance and outputs a summary of the current device state. The user instructions, the current device status summary, the multimodal perception information, and the structured instruction manual knowledge are input into the planning agent to infer and generate a phased operation plan; If a user command is found to lack necessary parameters or contain semantic ambiguity, a clarification exploration process is triggered to generate an exploratory operation sequence to guide user feedback. Based on the user feedback, the state acquisition and planning steps are re-executed until a complete operation plan sequence containing a description of the target control's appearance, location hints, and operation action text is output.

[0016] In one embodiment of the present invention, the feasibility and security assessment of the operation plan sequence using a reflective agent includes: The user instructions, historical execution information, multimodal perception information of the current round, the operation plan sequence, and the set of security constraints in the structured specification knowledge are input into the reflective agent; By reflecting on the intelligent agent, the risk and logical rationality of each step in the operation plan sequence are evaluated based on the set of safety constraints. If the operation plan is found to violate safety constraints or cannot be executed based on the current equipment status, the error reason and correction suggestions are output to guide the planning of intelligent weight planning. If the assessment of the operation plan is reasonable but there are situations where the device status is unknown or the functions of some buttons are uncertain, then an instruction to trigger the clarification and exploration process will be generated to guide the user to perform exploratory actions to obtain new panel perception information, and enter the next round of perception planning loop until the task is completed.

[0017] To achieve the above objectives, a second aspect of the present invention provides a device for understanding and generating operating instructions for home appliances based on multi-agent systems, comprising: The instruction manual summary agent is used to collect and parse multiple original instruction manuals of different models of home appliances before receiving user instructions. It extracts control names, function descriptions, operation steps, parameter ranges and safety constraint information from the instruction manuals, and generates category-level structured general instruction manuals based on semantic embedding and clustering, forming a general instruction manual knowledge base. A sensing agent is used to acquire multimodal sensing information of a target home appliance panel after receiving a user command. The multimodal sensing information includes the position, appearance features, and status readings of interactive components on the panel. The instruction manual understands and selects an intelligent agent, which determines the target instruction manual knowledge based on the multimodal perception information and user instructions through a hierarchical matching strategy, and maps the functional items in the target instruction manual knowledge to the interactive components in the multimodal perception information to obtain structured instruction manual knowledge; A network search agent is used to retrieve and extract missing operation steps and safety constraint information from the external network when the original instruction manual is missing, the matching degree is insufficient, or the coverage of general instruction manual knowledge is lower than the preset requirements, generate complete instruction manual knowledge, and return it to the instruction manual understanding and selection agent. A state acquisition agent is used to infer the current device state based on the structured specification knowledge and the multimodal perception information; A planning agent is used to take the user instructions, the structured specification knowledge, the multimodal perception information, and the current device status as inputs to generate an operation plan sequence that includes a description of the target control and operation actions; The reflective agent is used to evaluate the feasibility and safety of the operation plan sequence. If the evaluation is passed but the task is not completed, the user is guided to execute and enter the next round of perception planning loop. If the evaluation is not passed or there is uncertainty in the state, the clarification exploration process is triggered to correct the operation plan sequence until a complete operation step sequence is generated. The storage module is connected to the intelligent agent for understanding and selecting the manual, the intelligent agent for obtaining the state, the intelligent agent for planning, and the intelligent agent for reflecting, and is used to store the structured manual knowledge, historical execution information, and operation plan sequence.

[0018] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, for implementing a method for understanding and generating operating instructions for home appliances based on multiple agents as described in the first aspect embodiment.

[0019] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for understanding and generating operating instructions for home appliances based on multiple agents, as described in the first aspect embodiment.

[0020] The embodiments of the present invention have the following beneficial effects: This invention achieves the unification and reuse of cross-model control fields by building a universal instruction manual knowledge base before accepting instructions, reducing reliance on instruction manuals for specific models; it effectively solves the problem of missing instruction manuals or model mismatches through the first round of "panel perception - instruction manual selection / completion - structured mapping" process; through the closed-loop iterative mechanism of "perception - status acquisition - planning - reflection" in subsequent rounds, the system can continuously revise the operation plan based on the latest panel status; and through the clarification and exploration mechanism, it guides users to conduct low-cost verification at key uncertainties, thereby significantly improving the reliability, completeness, and security of the generated operation step sequence. Attached Figure Description

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a method for understanding and generating operation instructions for home appliances based on multi-agent systems, provided in an embodiment of the present invention; Figure 2 A framework diagram of a method for understanding and generating operation instructions for home appliances based on multi-agent systems, provided in an embodiment of the present invention; Figure 3 This is a structural diagram of a home appliance instruction manual understanding and operation command generation device based on multiple agents, provided in an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] The following describes, with reference to the accompanying drawings, a method for understanding and generating operation instructions for home appliances based on a multi-agent system, according to an embodiment of the present invention.

[0025] Example 1 This embodiment provides a method for understanding home appliance instruction manuals and generating operation instructions based on multi-agent systems, such as... Figure 1 and Figure 2 As shown, the method includes the following steps: S1. After receiving the user's instruction, acquire the multimodal perception information of the target home appliance panel. The multimodal perception information includes the position, appearance features and status readings of the interactive components on the panel.

[0026] S2, based on the multimodal perception information and user instructions, the target instruction manual knowledge is determined through a hierarchical matching strategy, and the functional items in the target instruction manual knowledge are mapped to the interactive components in the multimodal perception information to obtain structured instruction manual knowledge.

[0027] S3, input the user instructions, the structured specification knowledge, the multimodal perception information, and the current device status into the multi-agent system, and generate an operation plan sequence containing target control descriptions and operation actions through the planning agent; the multi-agent system includes at least a perception agent, a state acquisition agent, a planning agent, and a reflection agent.

[0028] S4. The reflective agent is used to evaluate the feasibility and safety of the operation plan sequence. If the evaluation is passed and the task is not completed, the user is guided to execute and enter the next round of perception planning loop. If the evaluation is not passed or there is uncertainty in the state, the clarification exploration process is triggered to correct the operation plan sequence until a complete operation plan sequence is generated.

[0029] Before performing the above steps, the present invention also includes a step of pre-constructing a general instruction manual knowledge base for the planning agent based on the instruction manual summary, so as to provide structured and searchable instruction manual knowledge for the planning agent even when only instruction manuals of similar categories are available later. The specific construction process is as follows: (1) Collect a set of original instruction manual documents D of various home appliance categories and different brands and models, and preprocess the instruction manuals; the preprocessing includes: converting PDF / web pages / scanned images into parsable text, and retaining information such as chapter levels, step lists, tables and warnings / precautions, so as to facilitate subsequent extraction.

[0030] (2) Extract a set of structured fields from each instruction manual; the fields shall include at least: control name and alias, control function description, typical operation steps, parameter range (temperature / time / power / gear, etc.), status variables (power supply / door status / mode / display reading, etc.) and safety constraints (warnings, prohibited operations, prerequisites, etc.).

[0031] (3) Semantic embedding and clustering are performed on the extracted control names to obtain a general control dictionary; preferably, the semantic embedding model uses all-MiniLM-L6-v2 to generate embedding vectors, and the clustering algorithm uses HDBSCAN to merge control names with similar semantics into the same cluster, and select the cluster center name as a unified name; at the same time, rule constraints are introduced to avoid obviously conflicting controls being merged incorrectly (e.g., keyword conflicts such as "up / down" and "start / stop").

[0032] (4) Merge and organize the multi-source descriptions within the same general control cluster to generate a category-level structured general instruction manual M and store it in the database; the M shall include at least: a set of function items, a set of step templates, a set of parameter ranges, a set of state variables and a set of safety constraints, and support retrieval by home appliance category and control semantics.

[0033]

[0034] Through the above steps, this embodiment completes the preparation of the general instruction manual knowledge base before the instruction arrives. Subsequently, in the first round of execution, the instruction manual understanding and selection agent will prioritize matching the original instruction manual; if the original instruction manual does not exist or the matching degree is insufficient, it will fall back to selecting similar category instruction manuals from the general instruction manual knowledge base; if the coverage is insufficient, a network search agent will be triggered to complete the information, thereby ensuring that the subsequent planning and reflection stages have sufficient instruction manual knowledge support.

[0035] Furthermore, this embodiment provides a detailed description of step S1: In this embodiment, obtaining the multimodal sensing information of the target home appliance panel in the t-th round includes: First, the image of the home appliance panel in round t is obtained. That is, the sensing agent takes a picture of the current home appliance panel through the camera and obtains a panel image. .

[0036] Next, visual and text recognition are performed on the panel image to obtain multiple interactive components and text information near the components; that is, the panel image is analyzed using an object detection model and an OCR text recognition model to identify the various control components on the panel. and adjacent signage .

[0037] Then, the multiple components and their corresponding text are input into a multimodal large model to obtain the possible functional descriptions for each component. The device display readings, indicator light statuses, etc., are also detected, and the overall sensory information of the entire home appliance panel is then obtained. That is, the components... and text The data is packaged and transmitted to a multimodal large model, which performs semantic analysis and outputs functional descriptions of each component on the appliance panel. It also identifies the current device's displayed values, indicator lights, and other status information. All the sensor information from the entire panel is encapsulated into a sensing package. .

[0038] Preferably, if there is an interactive dialogue interface or prompt information on the appliance panel, the sensing agent will also parse and extract the prompt content or user input information on the current interface. And store it. The encapsulation process of the panel perception information in round t can be represented as:

[0039] in, This represents the encapsulated panel perception information for round t. Through the above steps, the perception agent acquires comprehensive panel information, including the component list, component location, component text, component function description, display readings, and indicator light status.

[0040] Furthermore, this embodiment provides a detailed description of step S2: Specifically, the instruction manual explains that selecting an intelligent agent yields panel information provided by the perceptual intelligent agent. And user command I. The agent will use the information to match the corresponding instruction manual knowledge in order to provide a basis for subsequent planning. Specifically, this includes the following situations: If an original instruction manual that perfectly matches the target appliance model or function exists, the relevant content in that original manual is directly extracted to generate structured instruction manual knowledge. That is, the instruction manual understanding agent first queries the instruction manual database or knowledge base to find the instruction manual text that perfectly corresponds to the current device model. If a matching original instruction manual is found, its layout structure is analyzed and text is extracted, including identifying chapters, step lists, and precautions areas, extracting information related to user instructions and perception. The relevant operating procedures and parameter descriptions, as well as safety constraint information, form a structured instruction manual for this equipment.

[0041] If no original instruction manual directly matches the target appliance, or the matching degree is below a preset threshold, the most similar general instruction manual knowledge is selected from the general instruction manual knowledge base and applied to the appliance; that is, the instruction manual understanding agent calls the pre-built general instruction manual knowledge base to retrieve general instruction manual entries that are the same as or similar to the target appliance category (e.g., air conditioner, oven, etc.). Information is perceived through the calculation panel. The semantic similarity between the extracted component names and the functional part names in the general instruction manual is calculated (e.g., by generating embedding vectors using the pre-trained model all-MiniLM-L6-v2 and calculating cosine similarity). The most matching content in the general instruction manual is then used as the basic instruction manual knowledge for the device. This general instruction manual knowledge is then combined with the specific circumstances of the current device, aligning the control names (matching the control names / icons in the manual with the actual components on the panel) to form structured instruction manual knowledge applicable to the target home appliance.

[0042] If the coverage of parameters or specific controls required by the user's instructions is still lower than the preset requirements, a web search agent is invoked to retrieve external materials to supplement the instruction knowledge. That is, when the existing instruction content cannot fully guide the user to complete the task, the instruction understanding agent will construct a search query and invoke the web search agent to obtain relevant operating steps, guidelines, or safety tips from the internet. The web search agent performs a web keyword search through an online search engine API, uses web crawling tools to crawl candidate web pages or documents, and uses PyPDF2 to parse PDF instruction manuals or technical documents. From the acquired external materials, the agent extracts key steps, parameter descriptions, and safety constraint information relevant to the current instruction, supplementing them into the instruction knowledge to obtain a complete, structured instruction knowledge.

[0043] Through the above methods, the instruction manual understands and selects the intelligent agent to ultimately generate a structured instruction manual that matches the target home appliance and the current user's instructions. and store it in the memory module. Alternatively, it can be stored in a dedicated knowledge base for subsequent use by intelligent agents. The process of understanding and selecting the instruction manual can be represented as follows:

[0044] in, This indicates that the instruction manual explains the selection process; 'I' represents a user command. The information perceived by the current panel is output. This is structured instruction manual knowledge (which may include original instruction manual content, general instruction manual content, and content supplemented from the internet).

[0045] Furthermore, this embodiment provides a detailed description of step S3: Specifically, the planning agent acquires panel information provided by the sensing agent. User instructions I, and the structured instruction manual provided by the intelligent agent to understand and select the agent. Preferably, before commencing work, the planning agent also calls the state acquisition agent to obtain the current device state information. This assists in decision-making. The intelligent system analyzes the display readings, indicator light status, and historical operation results from the sensor information on the control panel to determine the current status or mode of the target appliance, such as whether the device is powered on, what the operating mode is, and what the temperature / time settings are. This status information is then incorporated into subsequent planning references. The device status acquisition process can be represented as follows:

[0046] in, This represents the state acquisition function, which takes the current sensor packet as input. Output the extracted status information .

[0047] In round t, since there is no historical execution data, the planning agent mainly generates operation steps based on current information without summarizing past actions. The user instructions I and the structured specification knowledge... Panel sensing information Status information The system inputs pre-stored additional prompts (such as general safety guidelines) into the large model as prompt words to generate the operation plan for round t. And the specific operating steps. The large model integrates memory modules. The stored current instructions and manual knowledge, and the panel information provided by the sensing agent. and status information Then, the inference agent outputs a specific action plan for the next step. This plan includes which operation to perform on which control of the appliance, and a textual description of that operation. The output of the planning agent includes not only the action plan for round t, but also... It can also include a summary of the steps that have been performed so far. (in the first round) (Empty or merely an understanding of the current instruction). The operation plan generation process can be represented as:

[0048] in, This is a memory information module (containing historical information such as the user's current commands, instruction manual knowledge, and executed steps). Based on the reflection results of the previous round (the first round has no results from the previous round, so it can be empty), the large model returns a summary of the actions taken in each previous step. and the next steps It should be noted that in the first round of planning, the planning agent only performs planning and does not need to summarize and update based on historical actions. Primarily used as a placeholder.

[0049] Planning the intelligent agent to generate action plans In addition, the process also includes: introducing a clarification mechanism to determine whether clarification of the current instruction or decision is needed. If the user instruction has a pre-defined problem, a clarification process is triggered before outputting the action plan, and the system directly proceeds to the next round of planning, awaiting user feedback. The pre-defined problems include the user instruction lacking necessary parameters and semantic ambiguity. That is, when the user-provided instruction is incomplete (e.g., lacking key parameters such as temperature or time) or vague, the planning agent will mark `clari_flag` as True, thus pausing the normal execution process and prompting the system to ask the user a clarification question. The clarification question is automatically generated by the system based on the missing information, such as prompting the user to supplement the required parameters or explaining the ambiguous instruction. After the user provides a clarification answer I', the system obtains new instruction information and jumps to the next round (i.e., re-enters step S1 to obtain new perception information or directly enters S3 to re-plan, depending on the specific implementation).

[0050] If the clarification process does not need to be activated in step S3 (clari_flag == False, clarification is invalid), then the output of the planning agent is... This will be directly submitted as the execution plan for this round of processing. For this invention, the action plan output by the planning agent is... In essence, this is a set of operational guidance steps for human users. In this embodiment, the planned operational steps are encapsulated as action descriptions and corresponding structured action instructions (e.g., represented by control identifiers and action types, for recording and subsequent reflection). The system then outputs these steps through the user interface to guide the user in performing the corresponding operations on the home appliances.

[0051] Furthermore, this embodiment provides a detailed description of step S4: When entering the reflective agent phase, the system has already acquired the panel perception information from round t. ) and the action plan for round t. The reflective agent inputs the above information, along with user instruction I and the historical action sequence (stored in the memory module Mt), into the large model, and outputs the reflection result Rt for the operation sequence of round t. Combining the knowledge from the instruction manual and safety constraints, it evaluates the plan for this round, summarizing the effects or identifying problems.

[0052] Through this reflection mechanism, the multi-agent system can promptly correct errors and adjust its strategies, ensuring the entire task progresses in the right direction. After each round of reflection, the memory information module... It will be updated accordingly.

[0053] Simultaneously, it determines whether the task corresponding to the current user instruction has been completed. Specifically, if the planning agent has output a complete sequence of operation steps, the current task ends; if there are undetermined subsequent operations, it continues to the next round of the perception-planning-reflection closed-loop process until the task is completed. Step S4 is a loop judgment module used to monitor task progress: if the target operation involved in user instruction I has been output, the process terminates; otherwise, the system prepares to obtain the perception information of the next round and proceed with subsequent steps. This judgment can be achieved by pre-determining the number of stages of task decomposition by the planning agent, or by evaluation by the reflection agent. The operation plan reflection evaluation process can be represented as:

[0054] In practical applications, for simple instruction tasks (such as operations that can be completed in one step), the completion conditions may be met in the first round, at which point the process ends directly. However, for complex instruction tasks (such as home appliances with many unknown states that need to be explored, requiring multiple interactive operations), multiple rounds of loops will be used to gradually guide the user to complete all steps.

[0055] Example 2 This invention provides a multimodal, multi-agent collaborative framework for guiding the automatic operation of home appliances. It includes modules for instruction manual understanding and selection, general instruction manual summarization, network search, perception, state acquisition, planning, and reflection, which, together with a memory module, decompose and execute complex tasks. Specifically: (1) The intelligent agent is selected to parse and match the knowledge in the instruction manual. When the instruction manual is missing or incomplete, it relies on the general knowledge base and network search to supplement the required information, thus ensuring the professionalism and completeness of the guidance steps. (2) General instruction manual: The intelligent agent uses the cluster analysis results of multiple instruction manuals to provide general functional instructions and operating specifications for various home appliances, which greatly improves the efficiency and reliability of knowledge acquisition in the absence of matching instruction manuals; (3) As an external information acquisition interface, the network search agent can extract the latest equipment operation guidelines and safety procedures from the Internet in real time, thus expanding the knowledge scope of the system; (4) The sensing agent uses computer vision and OCR technology to perform multimodal perception of home appliance panels and generates semantic descriptions with the help of large models, so that the system can “understand” the device interface and provide accurate environmental information for decision-making. (5) The state acquisition agent tracks and records the changes in device state and context information to assist in planning and decision-making and ensure that the operation sequence and preconditions are correct; (6) The planning agent reasoned about the existing information and knowledge, and generated the next operation plan in stages to ensure that the multi-step task was properly broken down. Each step included a description of the appearance of the specific control and a location prompt, so that the user could execute it directly. (7) The reflective agent verifies and corrects the planning results, and adjusts subsequent strategies in a timely manner when deviations occur, reducing the risk of misoperation. Through the division of labor and cooperation of the above-mentioned agents, the present invention can handle problems such as complex interfaces, difficulty in navigating multi-level menus, and incomplete manuals in home appliance operation, outputting reliable and clear textual operation steps, significantly reducing the cost for users to consult manuals and repeatedly try and fail, and improving the usability, explainability, and interactive performance of home appliance operation.

[0056] Based on the above embodiments, this embodiment provides a detailed description of the present invention through a complete process experiment: I. Deployment Phase: The specific deployment details are as follows: (1) The OCR text recognition model used cv_resnet18_ocr-detection to detect and recognize the button labels in the home appliance panel image; the object detection model used GroundingDINO to locate the functional controls and indicator icons on the panel; the image-text joint reasoning model used Qwen-vl-max and GPT-5 as a multimodal large model to undertake reasoning tasks such as semantic description of components, operation plan generation and human-computer dialogue.

[0057] (2) Construction of the General Manual Knowledge Base: Before receiving user instructions, the manual summarizing agent pre-collects and parses a large number of original manuals for different types and models of home appliances. The pre-trained language model all-MiniLM-L6-v2 is used to convert the names of functional components extracted from each manual into embedding vectors. The HDBSCAN clustering algorithm groups names with high semantic similarity into one category, and the name with the highest average semantic similarity in each cluster is selected as the name of that component. Descriptions of corresponding functions in each manual are merged to obtain a comprehensive description. Rules are introduced during the integration process to avoid incorrect merging (e.g., functions with "up / down" are not confused). Finally, the unified functional names, comprehensive descriptions, and basic equipment specifications of all clusters are summarized to form a draft general manual for that appliance category. After manual or large-scale model verification of its fluency and completeness, it is stored in the general manual knowledge base for subsequent use. This construction process can be represented as:

[0058] (3) Install and deploy related dependent environments: When implementing the method of this invention, it is necessary to install related dependent libraries and software components such as computer vision, natural language processing, and multi-agent framework. For example, configure a deep learning framework to load the above model, and install online search engine API interface library, Jina crawler tool, PyPDF2 parsing library, etc., to ensure that the functions of each module are called normally.

[0059] (4) Input the relevant API keys and configurations: After deployment, configure the keys and interface URLs of the online search engine API, large model and other required by the network search agent into the system, and ensure that the system has network access permissions so that it can call external search services to obtain supplementary information in the manual when needed.

[0060] II. Implementation Phase: The specific implementation details are as follows: (1) The user inputs the instruction I to be executed through the terminal interface. After receiving the user's instruction information, the system starts the multi-agent collaboration process.

[0061] (2) The sensing agent perceives the target home appliance panel: First, the camera is controlled to acquire the image of the current home appliance panel. The image is then input into cv_resnet18_ocr-detection to detect text regions on the panel, and simultaneously into the GroundingDINO object detection model to identify the position and category of each interactive control on the panel. The outputs of these two models yield the position and category identifiers of all interactive components on the panel, as well as text information near the components. and ).

[0062] (3) Pack the component set ICT and text set At obtained in step (2) into the multimodal large model Qwen-vl-max. The large model performs graphic semantic reasoning on each component to generate the functional description Dt of the component and the visual feature summary of the component. For example, the large model will output a description such as "Component 1 is a power switch button, red circle, located in the upper right corner of the panel, used to control the power of the device", and at the same time analyze the current display reading (such as the current temperature display) and the on / off status of the indicator lights on the current panel to determine whether the keyboard / touchpad is active and other status information.

[0063] (4) Summary of perceptual agents , , In addition to device status information, construct a panel sensing package. .this It contains complete perception information for the device panel in the current cycle. For example: component list and its location, text label for each component, function description, current displayed value, and indicator light status, etc.

[0064] (5) If there are user interaction dialog boxes or prompts on the panel (e.g., a smart screen prompting the user to confirm an option), the sensing agent in This interactive information will also be included. However, in this scenario, it is assumed that the home appliances only have physical buttons and no dialog boxes. Empty. The perceptual agent will ultimately obtain the panel perception package. The output is provided for use by subsequent intelligent agents, while the user's original instruction I is stored in the memory module. The encapsulation process of panel perception information in round t can be represented as:

[0065] Through the above steps, the system has completed the sensing of the appliance panel and obtained detailed panel information. .

[0066] (6) The instruction manual explains how to select and start the intelligent agent, utilizing the sensory intelligent agent's capabilities. The system matches and selects instruction manual knowledge based on user instruction I. Since this embodiment assumes that no original electronic instruction manual perfectly matches the specific device, the instruction manual understanding and selection agent first queries the general instruction manual knowledge base for the corresponding category. The agent compares... The main control names in the code were vectorized with the function names in a home appliance instruction manual from a general knowledge base. It was found that most controls (such as temperature knobs and mode selection switches) semantically corresponded. Therefore, a draft of a general instruction manual for this oven category was selected as the basic instruction manual knowledge, and preliminary structured instruction manual knowledge was generated based on this. Next, the instruction manual's intelligent agent detected a security warning related to the user's command, and that some information was missing from the general knowledge. Therefore, it decided to call the network search agent to supplement the relevant content.

[0067] (7) The web search agent constructs a query based on the current task. By calling the online search engine API, a web search is performed, yielding several potentially useful webpage links, including kitchen forum Q&A and a PDF instruction manual for a specific brand of appliance. The agent then uses a web crawler to open one of the higher-ranking PDF instruction manuals, parses its content using PyPDF2, and extracts the safety precautions section related to the instruction. This supplementary information is added to the structured instruction manual knowledge, refining the steps in the original general instructions to form the final version. The selection process, as explained in the instruction manual, can be represented as follows:

[0068] At this point, the system has acquired complete instruction manual knowledge for user commands, including the required operating steps, related parameters, and precautions.

[0069] (8) The state acquisition agent analyzes the current device state based on the perception package. The agent detects that the appliance is currently off (e.g., the power indicator is off, and the temperature display shows no reading). This state information is important for subsequent planning. The agent acquires the state information. Record it, for example It includes fields such as: {e.g., Power: OFF, Current Temperature: Empty, Door Status: Closed}, and provides them to the planning agent for reference. The device status acquisition process can be represented as:

[0070] (9) Enter the planning phase of round t (i.e., round 1). The planning agent will use the user instructions I, structured specification knowledge... Current panel sensing information and status information and memory modules Additional hints stored in the database are input into the large model for reasoning. Since this is the first execution round, historical actions and reflection information are empty; the planning agent at this stage primarily generates an initial plan based on the instruction manual. The large model analysis reveals that to complete the instruction, it needs to explore the current state of the appliance, for example, if it's not currently powered on, the user needs to turn it on. Only after sensing the appliance's state can the planning agent output the completed plan. The planning agent will then output the specific operation plan. The process of generating an operation plan can be represented as follows:

[0071] The planning agent obtained the plan in this round It actually contains a specific executable operation (pressing the power button), so it is directly broken down into an action description and action instructions for execution.

[0072] (10) Since the user instruction parameters are clear in this round of planning, there is no need to wake up any clarification process (clari_flag ==False), and the system enters the reflection stage according to the planning results.

[0073] (11) Panel perception information before and after execution And the action plan for this round. User instructions I and memory module The input reflects the intelligent agent, generating the reflection result Rt for this round of operations.

[0074] In this embodiment, the reflective agent evaluates the plan. The reflective agent returns... This indicates that the current plan is correct and no correction is needed. (Reflection results) The `err_flag` flag is set to `False` (indicating no errors need correction). If the reflective agent deems the plan complete at this point, it can send the plan to the user and terminate the process. The operational plan reflection and evaluation process can be represented as:

[0075] (12) If the reflection results indicate that this round of planning is an exploratory process, further updates are needed, and the process proceeds to the next round. At this point, Record to memory module In this process, the second cycle of perception-planning-execution-reflection begins. In the current case, further exploration is needed, so the process continues.

[0076] III. The second round begins.

[0077] (1) The sensing agent senses the home appliance panel again and obtains the panel information in the (t+1)th round (second round). At this point, the power button has been pressed by the user, and the device is powered on. Therefore, the power indicator light in the new panel image lights up, and the display screen begins to show the current status reading of the home appliance. The sensing agent repeats the process from (2) to (4): acquiring screenshots. Identify controls and text Describe the component's functionality And update the device status information (power indicator light changes from OFF to ON, current appliance status reading). Encapsulate the updated information into a new sensing package. .

[0078] (2) Display screen information Results of the previous round of reflection The updated memory information is formed by combining historical planning, user commands, etc. Input the planning agent.

[0079] (3) The planning agent summarizes the previous actions in the second round and plans the next action. The large model is based on the memory module. The content is summarized first. For example, "Power is on." Then refer to the instruction manual for further information. Determine if a complete action plan can be output next. Since the previous round of reflection indicates the power is on and there are no errors to correct, the planning agent continues according to the normal steps. Therefore, the action plan is output. Since all the information about the home appliances has been obtained, this round can directly output the complete planning sequence. If there are still some hidden states that need to be confirmed, this round will continue to output exploration steps.

[0080] (4) In this round, the current interface does not require the user to make multiple selections, and the user's instructions are clear, therefore clari_flag == False. The system directly sends the steps output by the planning agent to the user for execution through the interface.

[0081] (5) Including this round of action plans ,memory Input the reflection agent and obtain the reflection result Rt+1.

[0082] The reflecting agent believes that the plan is correct and that the plan has output complete operational steps, so the task can be terminated.

[0083] (6) The task corresponding to this user instruction I has ended. If the user sends a new instruction I', the system will continue to execute, start a new round of multi-agent collaboration, and generate operation instructions for the new task.

[0084] Example 3 This invention also provides a device for understanding and generating operating instructions for home appliances based on multi-agent systems, such as... Figure 3 As shown, the device 10 includes: The instruction manual summarizes the intelligent agent 100, which is used to collect and parse multiple original instruction manuals of different models of home appliances before receiving user instructions. It extracts control names, function descriptions, operation steps, parameter ranges and safety constraint information from the instruction manuals, and generates category-level structured general instruction manuals based on semantic embedding and clustering, forming a general instruction manual knowledge base. The sensing agent 200 is used to acquire multimodal sensing information of the target home appliance panel after receiving a user instruction. The multimodal sensing information includes the position, appearance features and status readings of interactive components on the panel. The instruction manual understanding selects intelligent agent 300, which is used to determine the target instruction manual knowledge through a hierarchical matching strategy based on the multimodal perception information and user instructions, and to map the functional items in the target instruction manual knowledge to the interactive components in the multimodal perception information to obtain structured instruction manual knowledge; The network search agent 400 is used to retrieve and extract missing operation steps and safety constraint information from the external network when the original instruction manual is missing, the matching degree is insufficient, or the coverage of general instruction manual knowledge is lower than the preset requirements, generate complete instruction manual knowledge, and return it to the instruction manual understanding and selection agent. A state acquisition agent 500 is used to infer the current device state based on the structured specification knowledge and the multimodal perception information; A planning intelligent agent 600 is used to take the user instructions, the structured specification knowledge, the multimodal perception information, and the current device status as inputs to generate an operation plan sequence that includes a description of the target control and operation actions; The reflective agent 700 is used to evaluate the feasibility and safety of the operation plan sequence. If the evaluation is passed and the task is not completed, the user is guided to execute and enter the next round of perception planning loop. If the evaluation is not passed or there is uncertainty in the state, the clarification exploration process is triggered to correct the operation plan sequence until a complete operation step sequence is generated. The storage module 800 is connected to the intelligent agent for understanding and selecting the instruction manual, the intelligent agent for obtaining the state, the intelligent agent for planning, and the intelligent agent for reflecting, and is used to store the structured instruction manual knowledge, historical execution information, and operation plan sequence.

[0085] Example 4 To implement the methods of the above embodiments, the present invention also provides a computer device, which includes a memory and a processor; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, so as to implement the various steps of the methods described above.

[0086] Example 5 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0088] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0089] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for understanding and generating operating instructions for home appliances based on multi-agent systems, characterized in that, Includes the following steps: After receiving a user instruction, the system acquires multimodal sensing information of the target home appliance panel, which includes the position, appearance features, and status readings of interactive components on the panel. Based on the multimodal perception information and user instructions, the target instruction manual knowledge is determined through a hierarchical matching strategy, and the functional items in the target instruction manual knowledge are mapped to the interactive components in the multimodal perception information to obtain structured instruction manual knowledge. The user instructions, the structured specification knowledge, the multimodal perception information, and the current device status are input into the multi-agent system, and the planning agent generates an operation plan sequence containing a description of the target control and operation actions; the multi-agent system includes at least a perception agent, a status acquisition agent, a planning agent, and a reflection agent; The feasibility and safety of the operation plan sequence are evaluated by the reflective agent. If the evaluation is passed but the task is not completed, the user is guided to execute and enter the next round of perception planning loop. If the evaluation is not passed or there is uncertainty in the state, the clarification exploration process is triggered to correct the operation plan sequence until a complete operation plan sequence is generated.

2. The method according to claim 1, characterized in that, The acquisition of multimodal sensing information of the target home appliance panel includes: Collect image data of the target home appliance panel, and use the target detection model to identify the bounding box coordinates and component category labels of all interactive components on the panel; Use text recognition to extract the label text and display screen readings in the vicinity of the component; The bounding box coordinates and component category labels are spatially associated and mapped with the identification text and display screen readings. The associated recognition results are then input into a multimodal large model for joint semantic reasoning of text and images to generate encapsulated perception information containing functional descriptions, appearance features, and indicator light status of each component, which serves as the multimodal perception information.

3. The method according to claim 1, characterized in that, Before receiving user instructions, build a general instruction knowledge base, including: Collect multiple original instruction manuals for different models of home appliances and extract the names and descriptions of their functional components; The extracted functional component names are mapped to semantic embedding vectors using an embedding model, and clustering algorithms are used to group names with semantic similarity higher than a set threshold into the same functional component cluster. For each functional component cluster, the name with the highest average semantic similarity within the cluster is selected as the unified name. The descriptions of the function in each instruction manual are merged and conflicting information is removed to construct the general instruction manual knowledge base.

4. The method according to claim 3, characterized in that, The method of determining the target specification knowledge through hierarchical matching strategy includes: The system queries the original instruction manual library and calculates the semantic similarity between the component names in the multimodal perception information and the instruction manual control fields in the library. If there is a model matching instruction manual with a similarity higher than a preset threshold, it is directly selected as the original instruction manual. If no matching instruction manual is available or the highest similarity is below a preset threshold, the general instruction manual knowledge base is invoked. The component name is converted into an embedding vector using a language model, and the similarity with the part name corresponding to each entry in the library is calculated. The general instruction manual entry with the highest similarity is selected as the basic instruction manual knowledge. If the basic instruction manual knowledge does not include the parameter range or specific controls involved in the user instructions, then search keywords are constructed to call external network resources for supplementary search, and the missing operation steps and safety constraint information are extracted from the search results and integrated to generate the final target instruction manual knowledge.

5. The method according to claim 1, characterized in that, The process of mapping functional items in the target specification knowledge to interactive components in multimodal perception information to obtain structured specification knowledge includes: Analyze the layout structure of the target instruction manual and extract the control names, icon features, operation steps, parameter ranges, and security constraints. Calculate the semantic similarity and visual feature matching degree between the extracted control names and icon features and the interactive components in the multimodal perception information, and establish the correspondence between the manual controls and the panel physical components. Based on the aforementioned correspondence, the functional items, operation steps, parameter ranges, and safety constraints in the instruction manual knowledge are associated with the corresponding interactive components to generate structured instruction manual knowledge containing preconditions and postconditions.

6. The method according to claim 1, characterized in that, The step of generating an operation plan sequence containing a description of the target control and operation actions through a planning agent includes: The state acquisition agent, based on the structured specification knowledge and multimodal perception information, infers the observable and unobservable state variables of the current home appliance and outputs a summary of the current device state. The user instructions, the current device status summary, the multimodal perception information, and the structured instruction manual knowledge are input into the planning agent to infer and generate a phased operation plan; If a user command is found to lack necessary parameters or contain semantic ambiguity, a clarification exploration process is triggered to generate an exploratory operation sequence to guide user feedback. Based on the user feedback, the state acquisition and planning steps are re-executed until a complete operation plan sequence containing a description of the target control's appearance, location hints, and operation action text is output.

7. The method according to claim 1, characterized in that, The feasibility and safety assessment of the operation plan sequence using a reflective agent includes: The user instructions, historical execution information, multimodal perception information of the current round, the operation plan sequence, and the set of security constraints in the structured specification knowledge are input into the reflective agent; By reflecting on the intelligent agent, the risk and logical rationality of each step in the operation plan sequence are evaluated based on the set of safety constraints. If the operation plan is found to violate safety constraints or cannot be executed based on the current equipment status, the error reason and correction suggestions are output to guide the planning of intelligent weight planning. If the assessment of the operation plan is reasonable but there are situations where the device status is unknown or the functions of some buttons are uncertain, then an instruction to trigger the clarification and exploration process will be generated to guide the user to perform exploratory actions to obtain new panel perception information, and enter the next round of perception planning loop until the task is completed.

8. A device for understanding and generating operating instructions for home appliances based on multi-agent intelligence, characterized in that, include: The instruction manual summary agent is used to collect and parse multiple original instruction manuals of different models of home appliances before receiving user instructions. It extracts control names, function descriptions, operation steps, parameter ranges and safety constraint information from the instruction manuals, and generates category-level structured general instruction manuals based on semantic embedding and clustering, forming a general instruction manual knowledge base. A sensing agent is used to acquire multimodal sensing information of a target home appliance panel after receiving a user command. The multimodal sensing information includes the position, appearance features, and status readings of interactive components on the panel. The instruction manual understands and selects an intelligent agent, which determines the target instruction manual knowledge based on the multimodal perception information and user instructions through a hierarchical matching strategy, and maps the functional items in the target instruction manual knowledge to the interactive components in the multimodal perception information to obtain structured instruction manual knowledge; A network search agent is used to retrieve and extract missing operation steps and safety constraint information from the external network when the original instruction manual is missing, the matching degree is insufficient, or the coverage of general instruction manual knowledge is lower than the preset requirements, generate complete instruction manual knowledge, and return it to the instruction manual understanding and selection agent. A state acquisition agent is used to infer the current device state based on the structured specification knowledge and the multimodal perception information; A planning agent is used to take the user instructions, the structured specification knowledge, the multimodal perception information, and the current device status as inputs to generate an operation plan sequence that includes a description of the target control and operation actions; The reflective agent is used to evaluate the feasibility and safety of the operation plan sequence. If the evaluation is passed but the task is not completed, the user is guided to execute and enter the next round of perception planning loop. If the evaluation is not passed or there is uncertainty in the state, the clarification exploration process is triggered to correct the operation plan sequence until a complete operation plan sequence is generated. The storage module is connected to the intelligent agent for understanding and selecting the manual, the intelligent agent for obtaining the state, the intelligent agent for planning, and the intelligent agent for reflecting, and is used to store the structured manual knowledge, historical execution information, and operation plan sequence.

9. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method for understanding and generating operation instructions for home appliances based on multiple agents as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a method for understanding and generating operation instructions for home appliances based on multiple agents, as described in any one of claims 1-7.

Images