A smart home control method and computer program product

Abstract

Embodiments of the present application provide a smart home control method and computer program product, relating to the technical field of large model. According to the obtained kitchen multi-modal sensor data, the current home environment state is identified, when the home environment state meets the preset trigger condition, a virtual interaction instruction is generated, according to the user interaction instruction and / or the virtual interaction instruction, as a task request to be executed, in response to the task request, based on the standardized device calling protocol, the large language model is used to decompose the task request into at least one subtask, and each subtask is mapped to a deterministic device control sequence for the target device, and the target device is controlled to perform the corresponding operation. Therefore, unified scheduling and active response to multi-source instructions are realized, and the active service capability of the smart home and the flexibility of cross-brand device collaboration are improved.

Description

Technical Field

[0001] This invention relates to the field of large-scale model technology, and more specifically, to a smart home control method and computer program product. Background Technology

[0002] With the development of smart home technology, controlling multiple devices to work together through natural language has become an important direction for smart homes.

[0003] However, existing smart home device control technologies still have the following shortcomings: On the one hand, existing systems rely solely on user voice commands as the only trigger source, and the system is always in a passive response state. When there are risk scenarios such as abnormal oil temperature or gas leaks in the kitchen, if the user does not actively issue a command, the system cannot autonomously identify the environmental state and actively intervene. On the other hand, smart homes involve a variety of devices such as lights, air conditioners, curtains, security sensors, kitchen appliances, and cleaning robots. Different brands and types of devices are heterogeneous in terms of communication protocols, control interfaces, and capability descriptions. Existing systems usually use hard-coded adaptation logic for fixed device combinations. When users issue complex tasks involving the collaboration of multiple devices, the system has difficulty dynamically discovering device capabilities, automatically orchestrating and allocating sub-tasks, and cross-brand collaboration has almost no scalability. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a smart home control method and computer program product to at least partially improve the above-mentioned problems.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, embodiments of the present invention provide a smart home control method, comprising: Based on the acquired multimodal sensor data of the home environment, the current state of the home environment is identified. When the state of the home environment meets the preset triggering conditions, a virtual interaction command with the same format as the user interaction command is generated. Based on the user interaction instructions and / or the virtual interaction instructions, a task request to be executed is made. In response to the task request, based on a standardized device invocation protocol, the task request is decomposed into at least one subtask using a large language model, and each subtask is mapped to a deterministic device control sequence for the target device. The target device is controlled to perform corresponding operations according to the device control sequence.

[0006] Optionally, the step of identifying the current home environment state based on the acquired multimodal sensor data of the home environment, and generating a virtual interaction command with a format consistent with the user interaction command when the home environment state meets a preset trigger condition, includes: Continuously collect multimodal sensing data through multiple sensors deployed within the home environment; Based on the multimodal sensing data, the current home environment status is identified through a state machine; When the current home environment state meets the preset triggering conditions, obtain the semantic understanding result of the current home environment state; Based on the semantic understanding results and the preset scene action mapping rules, virtual interaction commands with the same format as user interaction commands are generated.

[0007] Optionally, in response to the task request, based on a standardized device invocation protocol, the task request is decomposed into at least one subtask using a large language model, and each subtask is mapped to a deterministic device control sequence for the target device, including: In response to the task request, a real-time list of currently available device functions is obtained, and the task request is decomposed into at least one sub-task using a large language model; wherein each sub-task is associated with a specific capability of the device. Based on a standardized device invocation protocol, an abstract control instruction with parameter placeholders is generated for each of the subtasks; The parameter placeholders in the abstract control command are mapped to specific physical parameters to be executed for the target device, thereby generating a deterministic device control sequence.

[0008] Optionally, in response to the task request, obtaining the real-time function list of currently available devices and using a large language model to decompose the task request into at least one sub-task includes: Obtain the device identifier, device type, capability description, and operating status of each device currently connected to the system, and generate the real-time function list; The real-time function list and the task request are used to construct sub-tasks to extract prompt words; Based on the extracted prompt words from the subtasks, the task request is broken down into at least one subtask using a large language model; each subtask is associated with a specific capability of a device in the real-time function list.

[0009] Optionally, the step of generating abstract control instructions with parameter placeholders for each subtask based on a standardized device invocation protocol includes: Obtain a list of tools registered by each device that has connected to the standardized device invocation protocol; wherein each tool includes a tool name, function description and input parameter structure; The tool list and each of the subtasks are constructed into tool call prompts; Based on the tool call prompt words, the target tool corresponding to each subtask is matched from the tool list using the large language model; Based on the input parameter structure of the target tool, an abstract control instruction is generated containing the names of each parameter in the input parameter structure as placeholders.

[0010] Optionally, the step of mapping the parameter placeholders in the abstract control command to specific physical parameters executed for the target device to generate a deterministic device control sequence includes: Obtain the user preference parameters corresponding to each of the subtasks; Based on the user preference parameters, determine the physical parameter values ​​corresponding to each of the parameter placeholders; The physical parameter values ​​are filled into the corresponding parameter placeholders in the abstract control command to generate a deterministic device control sequence.

[0011] Optionally, obtaining the user preference parameters corresponding to each of the subtasks includes: The task request is semantically matched with the task layer nodes in the dynamic memory graph, and / or the subtask is semantically matched with the subtask layer nodes in the dynamic memory graph to determine the hit node; the task layer node contains a task semantic vector and its associated subtask identifier, the subtask layer node contains a device type, operation template and its associated context identifier, the dynamic memory graph also includes a context layer node, the context layer node contains a preference attribute description; When there are multiple hit nodes in the semantic matching result, obtain the weight parameters of each hit node, and select the node with the highest weight parameter as the target hit node; The corresponding context layer node is determined based on the context identifier associated with the target hit node; The preference attribute description is extracted from the context layer node and used as the user preference parameter.

[0012] Optionally, the method further includes: After the target device completes the operation, image data of the operation result is acquired by an image sensor; The image data is input into a visual language model for effect evaluation, and evaluation results are generated. Based on the evaluation results, update the weight parameters of the hit node; and / or, based on the evaluation results, update the mapping relationship in the context layer nodes associated with the hit node.

[0013] Optionally, the method further includes: The device control sequence is sent to a virtual sandbox for simulated execution; the virtual sandbox includes digital twin models of each device. When the simulation execution returns an error message, the error message is re-input into the large language model; The parameters associated with the error message in the device control sequence are adjusted using the large language model until the simulation is successful or the preset retry limit is reached.

[0014] Secondly, embodiments of the present invention provide a computer program product, the computer program product including instructions, which, when executed on an electronic device, cause the electronic device to perform the method described in any of the above-mentioned embodiments.

[0015] This invention provides a smart home control method and computer program product that continuously collects multimodal sensor data of the home environment and identifies environmental conditions. When risk scenarios such as abnormal temperature, gas leaks, or illegal intrusion are detected, the system can autonomously generate virtual interaction commands with a unified user interaction command format. Environmental events are incorporated into the scheduling process in a standardized form, thereby proactively initiating intervention when the user does not actively issue commands. In response to task requests, based on a standardized device invocation protocol, the system uses a large language model to decompose task requests into at least one sub-task and maps each sub-task to a deterministic device control sequence for the target device. This enables dynamic discovery of device capabilities, automatic arrangement and allocation of sub-tasks, and access and invocation of devices of different brands and types according to a unified protocol, improving cross-brand compatibility in smart home multi-device collaborative scenarios.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic structural block diagram of an electronic device provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a smart home control method provided in an embodiment of the present invention; Figure 3 A flowchart illustrating step S21 provided in an embodiment of the present invention; Figure 4 This is another schematic diagram of a smart home control method provided in an embodiment of the present invention; Figure 5 A flowchart illustrating step S231 provided in an embodiment of the present invention; Figure 6 A flowchart illustrating step S232 provided in an embodiment of the present invention; Figure 7 This is a flowchart illustrating step S233 as provided in an embodiment of the present invention.

[0019] Icons: 100 - Electronic device; 101 - Memory; 102 - Communication interface; 103 - Processor; 104 - Communication bus. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0022] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0024] In existing smart home device control technologies, systems typically rely solely on user voice commands as the only trigger source, leaving the system in a passive response state. For example, when a risk scenario occurs in the kitchen, such as abnormal oil temperature or gas leak, the system cannot autonomously identify the environmental state and proactively intervene unless the user issues a command.

[0025] Furthermore, smart homes involve a variety of devices such as lighting, air conditioning, curtains, security sensors, kitchen appliances, and cleaning robots. Different brands and types of devices are heterogeneous in terms of communication protocols, control interfaces, and capability descriptions. Existing systems typically use hard-coded adaptation logic for fixed device combinations. When users issue complex tasks involving the collaboration of multiple devices, the system struggles to dynamically discover device capabilities, automatically orchestrate and allocate sub-tasks, and cross-brand collaboration has virtually no scalability.

[0026] Based on the above, embodiments of the present invention provide a smart home control method and computer program product. By actively sensing the state of the home environment and generating virtual interactive instructions in a unified format, the method can uniformly schedule and process multi-source instructions and transform fuzzy natural language intentions into deterministic cross-device collaborative control sequences, thus solving the problems of passive system interaction and poor cross-brand collaborative scalability in the prior art.

[0027] To implement the process steps and functions of the various examples of this invention, please refer to [link / reference]. Figure 1 , Figure 1 This is a schematic structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, which are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101, thereby performing various functional applications and data processing.

[0028] Electronic device 100 can be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understood that electronic device 100 is not limited to a physical server, but can also be a virtual machine on a physical server, a virtual machine built on a cloud platform, or any other computer that can provide the same functionality as the server or virtual machine. The operating system of electronic device 100 can be, but is not limited to, Windows, Linux, etc.

[0029] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0030] The communication connection between the electronic device 100 and external devices is achieved through at least one communication interface 102 (which can be wired or wireless).

[0031] Processor 103 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of this embodiment can be completed by integrated logic circuits in the hardware of processor 103 or by instructions in software form. Processor 103 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0032] Understandable. Figure 1 The structure shown is for illustrative purposes only; the electronic device 100 may also include components that are more advanced than those shown. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0033] The following is an exemplary description of the smart home control method provided by this invention. For ease of understanding, this embodiment uses a kitchen scenario as an example; however, those skilled in the art will understand that the method provided by this invention is also applicable to whole-house smart home scenarios such as living rooms, bedrooms, and security systems. The smart home control method provided by this embodiment can be applied to a smart kitchen system, which may include various sensors deployed in the kitchen environment, at least one controllable kitchen appliance, and a computing processing unit. Sensors include, but are not limited to, cameras, infrared array sensors, gas sensors, etc. Kitchen appliances include, but are not limited to, ovens, stoves, food processors, range hoods, weighing scales, mixers, etc. The computing processing unit may be an electronic device 100, used to execute the smart home control method provided by this embodiment. See also Figure 2 The method includes, for example Figure 2 The following steps are described: S21: Based on the acquired multimodal sensor data of the home environment, identify the current state of the home environment. When the state of the home environment meets the preset trigger conditions, generate a virtual interaction command that is consistent with the user's interaction command format.

[0034] S22: As a task request to be executed based on user interaction instructions and / or virtual interaction instructions.

[0035] S23: In response to a task request, based on a standardized device invocation protocol, the task request is broken down into at least one sub-task using a large language model, and each sub-task is mapped to a deterministic device control sequence for the target device.

[0036] S24: Control the target device to perform the corresponding operation according to the device control sequence.

[0037] This method continuously collects multimodal sensor data of the home environment and identifies environmental conditions. When an abnormal or risky scenario is detected, it autonomously generates virtual interaction commands with a format consistent with user interaction commands. The user interaction commands and virtual interaction commands are then treated as a unified task request for scheduling and processing. In response to the task request, based on a standardized device invocation protocol, the method uses a large language model to decompose the task request into at least one sub-task and maps each sub-task to a deterministic device control sequence for the target device. This achieves unified scheduling and proactive response to multi-source commands and transforms fuzzy natural language intents into deterministic cross-device collaborative control sequences, thereby improving the proactive service capabilities of smart homes and the flexibility of cross-brand device collaboration.

[0038] In step S21, multimodal sensing data is continuously collected using various sensors deployed throughout the kitchen. This multimodal sensing data includes, but is not limited to: image data collected by cameras, temperature distribution data collected by infrared array sensors, and oil fume concentration and / or gas concentration data collected by gas sensors. The multimodal sensing data is fused into an environmental state vector, essentially a "health snapshot" of the kitchen.

[0039] The current home environment status is identified through a multimodal state machine. The state machine predefines basic kitchen state categories, including but not limited to idle, cooking, risky, and post-processing states. During each heartbeat cycle (e.g., once per second), the system compares the currently collected environmental state vector with the characteristic thresholds for each state defined in the state machine, autonomously determining whether the current environment is abnormal or poses a potential risk. Once the determination meets preset trigger conditions (e.g., indoor temperature exceeding a preset safety threshold, rapid increase in gas concentration, or doors and windows being opened while armed), the system immediately triggers an alert.

[0040] It's important to note that the heartbeat mechanism's "alertness" goes beyond simply issuing alarm signals. When the triggering conditions are met, the system saves the current environmental state and issues a high-priority message. Simultaneously, it uses a visual language model to perform semantic analysis of this environmental state, generating a natural language description of the scene. Based on preset scene action mapping rules, it then generates a virtual interaction command that is identical in format to the user's actively issued user interaction command, and these virtual interaction commands are uniformly incorporated into the subsequent task scheduling process.

[0041] Thus, the heartbeat mechanism enables the system to autonomously initiate service requests without explicit user intervention. The system's operating mode has shifted from the passive "event-driven" mode of existing technologies, which only begins working after receiving user instructions, to a dual "time-driven + event-driven" mode, where the system actively senses the environment and makes judgments in each heartbeat cycle. This truly realizes the transformation from "passive response" to "proactive service."

[0042] The preset trigger conditions are pre-defined abnormal environmental conditions that require active system intervention, such as temperatures exceeding 280°C, rapid increases in cooking fume concentration, and excessive gas concentration. When the current home environment meets any of these trigger conditions, a virtual interaction command is generated. This virtual interaction command uses the exact same data format as user-inputted voice commands, such as natural language strings. In other words, regardless of whether the command originates from user-initiated voice input or system-triggered activation, its source is not distinguished in subsequent processing.

[0043] For example, when the system detects that the oil temperature is too high, the generated virtual interactive command is "The oil temperature is too high. Please adjust the firepower immediately and pay attention to safety." When the system detects a gas leak, the generated virtual interactive command is "Safety alarm! Gas leak detected. The gas supply has been shut off in an emergency."

[0044] In one alternative implementation, see [link to implementation details]. Figure 3 Step S21 may include the following sub-steps S211 to S215: S211: Continuously collects multimodal sensing data through a variety of sensors deployed in the home environment.

[0045] Multimodal sensing data is continuously collected through a variety of sensors deployed within the home environment. These sensors include cameras, infrared array sensors, and gas sensors, which respectively collect image data, temperature distribution data, cooking fume concentration data, and / or gas concentration data.

[0046] S212: Based on multimodal sensor data, identify the current home environment status through a state machine.

[0047] Based on the collected multimodal sensor data, a state machine is used to identify the current state of the home environment. The state machine defines multiple preset state categories (such as idle, cooking, at risk, post-processing, etc.). The current multimodal sensor data is compared with the feature thresholds corresponding to each category to determine the current state category.

[0048] S213: When the current home environment status meets the preset trigger conditions, obtain the semantic understanding result of the current home environment status.

[0049] The system compares the current home environment with preset thresholds to determine if preset trigger conditions are met. These preset thresholds include, but are not limited to, temperature thresholds, cooking fume concentration thresholds, and gas concentration thresholds. For example, a temperature exceeding 280°C is considered an overheating risk, and a gas concentration exceeding a safety threshold is considered a gas leak risk.

[0050] When preset trigger conditions are met, the semantic understanding result of the current home environment status is obtained. This semantic understanding result is a natural language description of the current environmental status, such as "Deep frying is currently underway, and the oil temperature has exceeded the safety threshold" or "The flame on the left stove has been extinguished, and the gas concentration has increased."

[0051] Optionally, S213 may include: Input multimodal sensor data into the visual language model.

[0052] By using a visual language model, multimodal sensor data is transformed into a scene semantic description in natural language form, which serves as the result of semantic understanding.

[0053] Multimodal sensor data is input into the Visual Language Model (VLM). A Visual Language Model is a deep learning model that can process both visual input (images) and language input simultaneously, and has the ability to transform visual information into natural language descriptions.

[0054] By using a visual language model, multimodal sensor data (especially image data, as well as fused temperature data, gas data, etc.) are transformed into scene semantic descriptions in natural language form, which serve as semantic understanding results.

[0055] S214: Based on the semantic understanding results and the preset scene action mapping rules, generate virtual interaction commands that are consistent with the user interaction command format.

[0056] Based on semantic understanding results and pre-defined scene-action mapping rules, virtual interaction commands are generated that are consistent with the user's interaction command format. The scene-action mapping rule library pre-stores the mapping relationships between various scenes and corresponding response actions. For example, the "oil temperature too high" scene is mapped to "generate oil temperature warning reminder command", and the "gas leak" scene is mapped to "generate command to close the gas valve and open the exhaust fan". The generated virtual interaction commands are in natural language string format, completely consistent with the user interaction command format, ensuring that subsequent scheduling processes do not need to distinguish the source of the command.

[0057] Next, step S22 is executed to receive user interaction commands and / or virtual interaction commands. User interaction commands are voice control commands actively issued by the user, such as "Bake a cake for me" or "Turn down the stove." Virtual interaction commands are commands automatically generated by the system in step S21.

[0058] Because user interaction commands and virtual interaction commands use a unified data format, they can both be included in the same scheduling queue as task requests to be executed. "And / or" indicates three scenarios: only user interaction commands exist, only virtual interaction commands exist, or both user interaction commands and virtual interaction commands exist simultaneously. Regardless of the scenario, the system treats all received commands as task requests for subsequent processing in a uniform manner, without needing to distinguish the source of the commands at the scheduling layer. This achieves a unified scheduling architecture that grants equal rights to both humans and machines, reducing the internal processing complexity of the system.

[0059] After receiving the task request to be executed, the system enters the scheduling and execution phase, proceeding to step S23. The core objective of this step is to respond to the task request by breaking it down into at least one sub-task based on a standardized device invocation protocol and utilizing a large language model. Each sub-task is then mapped to a deterministic device control sequence for the target device. This step is not a simple instruction translation, but rather a transformation from open natural language intents to closed, executable control instructions. Through this step, the system can dynamically discover device capabilities, automatically orchestrate and allocate sub-tasks, enabling devices of different brands and types to access and invoke the system according to a unified protocol, thereby transforming ambiguous natural language intents into deterministic cross-device collaborative control sequences.

[0060] This step is implemented using a Decompose-Derive-Refine (DDR) inference pipeline. See [link / reference] Figure 4 It can include the following three sub-steps: S231 (Decomposition Phase): In response to the task request, obtain the real-time list of available device functions, and use a large language model to decompose the task request into at least one sub-task; where each sub-task is associated with a specific capability of a device.

[0061] S232 (Derivation Stage): Based on the standardized device invocation protocol, generate abstract control instructions with parameter placeholders for each subtask.

[0062] The standardized device invocation protocol can be the Model Context Protocol (MCP), and the capabilities of each device are encapsulated as standardized plugins (i.e., tools) through adapters and registered with the MCP server.

[0063] S233 (Refinement Stage): Maps the parameter placeholders in the abstract control instructions to the specific physical parameters to be executed for the target device, generating a deterministic device control sequence.

[0064] By using the step-by-step convergence of the three-stage pipeline described above, open natural language intentions are gradually transformed into closed, executable control instructions. Compared to the scheme of generating a complete control sequence through one-time reasoning, this significantly reduces the probability of instruction errors caused by large language model illusions.

[0065] Optionally, during DDR pipeline execution, verified subtask nodes can be cached and reused to reduce the overhead of repetitive inference in the large language model. For example, when a subtask generated in the decomposition phase semantically matches a historically verified subtask node, the abstract control instructions and parameter mapping results corresponding to that subtask node can be directly reused, skipping the large language model inference process in the derivation and / or refinement phases. Only necessary parameters need to be locally adjusted to generate a deterministic device control sequence. As a result, in steady-state operation scenarios, the number of repetitive inferences for general operations is significantly reduced, and the call frequency and resource consumption of the large cloud model are reduced accordingly.

[0066] Therefore, the unified tool registration mechanism based on standardized device invocation protocols breaks down protocol barriers between devices of different brands, solving the "ecosystem silo" problem caused by protocol incompatibility between heterogeneous devices. Simultaneously, through the step-by-step convergence of the three-stage pipeline and the caching and reuse mechanism of sub-task nodes, repeated inference of common operations by LLM is avoided, effectively reducing response latency and cloud resource consumption. Specifically, in the decomposition stage, in response to the task request determined in step S22, step S231 is executed. In an optional implementation, see [link to implementation details]. Figure 5 S231 may include the following sub-steps: S2311: Obtain the device identifier, device type, capability description and operating status of each device currently connected to the system, and generate a real-time function list.

[0067] S2312: Construct real-time function lists and task requests into subtasks to extract prompts.

[0068] S2313: Based on subtask extraction of prompt words, the task request is broken down into at least one subtask through a large language model; each subtask is associated with a specific capability of a device in the real-time function list.

[0069] The system retrieves the device ID (device_id), device type (type), capabilities description (capabilities), and operating status (status) of each device currently connected to the system, and generates a real-time function list. This function list can be in JSON format and includes all currently online callable devices and their supported operational capabilities. For example, the capability description of a weighing scale is "weighing (unit: grams)," and the capability description of an oven is "baking (temperature range: 50-250°C, mode: top and bottom heat / convection / steam)."

[0070] The real-time function list and task requests are used to construct subtasks for extracting prompts. These prompts contain explicit constraints requiring the large language model to use only the devices and capabilities listed in the real-time function list for task decomposition, and not to use devices or capabilities outside the list. For example, the prompt could be: "You are a kitchen planning assistant. You must and can only use the following device list for task decomposition: [Insert real-time function list]. Please decompose the instruction '[User Instruction]' into a series of subtasks, each subtask corresponding to a specific capability of a device in the list."

[0071] Based on the extracted prompt words from this subtask, the task request is broken down into at least one subtask using a large language model. Each subtask is associated with a specific capability of a device in the real-time feature list. The large language model outputs the breakdown results in JSON array format, with each subtask containing a device identifier (device_id) and an action description (action_description) field.

[0072] After acquiring each subtask, an abstract control instruction with parameter placeholders is generated for each subtask. In one optional implementation, see [link to implementation details]. Figure 6 Step S232 may include the following sub-steps: S2321: Obtain a list of tools registered by each device that has been connected to the standardized device invocation protocol; where each tool includes a tool name, function description and input parameter structure.

[0073] Obtain a list of tools registered by each device that has connected to the standardized device invocation protocol. Each device registers with the standardized device invocation protocol server through its adapter, acting as a tool provider. During registration, the following attributes of the tool must be defined strictly according to the protocol requirements: tool name, description, and input parameter structure. The input parameter structure is defined in JSON Schema format, containing information such as the name, data type, value range, and whether each parameter is required.

[0074] S2322: Construct the tool list and each subtask into tool call prompts.

[0075] The tool list and each subtask are constructed into a tool invocation prompt. This prompt allows the large language model to "see" all registered available tools and their invocation methods. For example, the prompt could be expressed as: "Based on the following subtask description, select the appropriate MCP tool and generate the invocation parameters. The available tool list is: [Insert Tool List]."

[0076] S2323: Based on tool call prompts, use a large language model to match the target tool corresponding to each subtask from the tool list.

[0077] Based on tool call prompts, a large language model is used to match the target tool for each subtask from the tool list. The matching method is semantic matching, that is, selection is based on the semantic similarity between the action description of the subtask and the name / description of the tool.

[0078] S2324: Generate abstract control instructions containing placeholders for each parameter name in the input parameter structure, based on the input parameter structure of the target tool.

[0079] Based on the input parameter structure of the target tool, abstract control instructions are generated, containing placeholders for the names of each parameter in the input parameter structure. For example, if the target tool is "bosch_oven_bake" and its input parameter structure contains two parameters, "temperature" and "duration", then the generated abstract control instruction would be: bosch_oven_bake({"temperature": "[temperature]", "duration": "[duration]"}). Here, [temperature] and [duration] are placeholders, indicating that the specific values ​​of these parameters will be determined and filled in subsequent steps. Using placeholders for unknown numerical values ​​eliminates the need for the large language model to guess specific values ​​during the derivation stage, thus avoiding errors caused by conjecture.

[0080] After obtaining the abstract control command, step S233 is executed to determine the specific values ​​of the placeholders within it. Mapping the parameter placeholders in the abstract control command to specific physical parameters can be achieved using various methods.

[0081] In one alternative implementation, parameter placeholders are mapped to device default parameters or industry-standard values. For example, for the "preheat oven" subtask, the temperature placeholder [temperature] in its abstract control instruction is matched with the device's default 180°C, and 180°C is directly entered. Alternatively, historical execution records for this device type can be retrieved, and the most frequently occurring historical parameter value can be used as the mapping result for this task.

[0082] In other implementations, parameter placeholders can also be filled using common-sense reasoning based on a large language model. For example, based on the common-sense concept of "baking a cake," the temperature placeholder [temperature] can be reasoned to be a specific value in the range of 180-200°C.

[0083] The determined physical parameter values ​​are filled into the corresponding parameter placeholders in the abstract control instructions to generate a deterministic device control sequence. For example, the abstract instruction `oven_bake({"temperature": "[temperature]", "duration": "[duration]"})` is filled into `oven_bake({"temperature": "210", "duration": "11"})`. When a task request is broken down into multiple subtasks, the deterministic control instructions corresponding to each subtask are organized into a device control sequence according to the execution order.

[0084] In another alternative implementation, see Figure 7 Step S233 may include the following sub-steps: S2331: Obtain the user preference parameters corresponding to each subtask.

[0085] Retrieve user preference parameters for each subtask. This retrieval can be achieved by requesting user preferences within the current context from the memory system. The memory system stores user preferences, task templates, and execution history in a directed acyclic graph structure. For example, if the retrieved user preference parameters are {"Crispness": "high", "Browning": "medium"}, it indicates that the user prefers a high level of crispness and a medium level of browning for the current task.

[0086] Optionally, semantic matching is performed between the task request and the task layer nodes in the dynamic memory graph, and / or semantic matching is performed between the subtask and the subtask layer nodes in the dynamic memory graph to determine the matching nodes. The steps for obtaining the user preference parameters corresponding to each subtask may include: S23311: Perform semantic matching between the task request and the task layer nodes in the dynamic memory graph, and / or perform semantic matching between the subtask and the subtask layer nodes in the dynamic memory graph to determine the hit node; the task layer node contains the task semantic vector and its associated subtask identifier, the subtask layer node contains the device type, operation template and its associated context identifier, the dynamic memory graph also includes context layer nodes, and the context layer nodes contain preference attribute descriptions.

[0087] S23312: When there are multiple hit nodes in the semantic matching result, obtain the weight parameters of each hit node and select the node with the highest weight parameter as the target hit node.

[0088] S23313: Determine the corresponding context layer node based on the context identifier associated with the target hit node.

[0089] S23314: Extract preference attribute descriptions from context layer nodes as user preference parameters.

[0090] The dynamic memory graph stores a user's historical preferences, task templates, and execution history in a directed acyclic graph (DAG) structure, comprising a three-layer node structure. Task-layer nodes contain task semantic vectors and their associated subtask identifiers, used to store the semantic information of the user's complete instructions; subtask-layer nodes contain device types, operation templates, and their associated context identifiers, used to store abstract templates for specific device operations; and context-layer nodes contain preference attribute descriptions, used to store the user's preference settings in a specific context.

[0091] It's important to note that the user preferences stored in the context layer nodes are not physical parameters specific to any particular device, but rather device-independent attribute descriptions after "environmental attribute abstraction." For example, a user's preference for steak doneness is not stored as a specific device's temperature value (e.g., "65°C"), but as a device-independent abstract attribute description (e.g., "doneness preference: medium well"). When execution is required on a specific target device, the abstract attribute is mapped to the corresponding physical parameters based on the device's type and capabilities. By decoupling user preferences from device physical parameters, users' personalized habits are no longer bound to any specific brand or model of device, thus laying the data foundation for cross-device migration.

[0092] The system calculates semantic similarity between the complete user-input command (task request) and the task semantic vector of the task layer nodes, and / or performs semantic matching between the decomposed subtask descriptions and the device type and operation template of the subtask layer nodes to determine the hit nodes. When multiple hit nodes are found in the semantic matching results, the weight parameters of each hit node are obtained, and the node with the highest weight parameter is selected as the target hit node. The value of the weight parameter reflects the frequency of access to the corresponding memory, the most recent access time, and the historical execution success rate; the higher the weight, the more preferentially the memory should be used.

[0093] Based on the context identifier associated with the target hit node, determine the corresponding context layer node. If the hit is a task layer node, locate the corresponding subtask layer node through the associated subtask identifier in the task layer node, and then locate the context layer node through the context identifier in the subtask layer node; if the hit is a subtask layer node, directly locate the context layer node through the context identifier.

[0094] Finally, preference attribute descriptions are extracted from the context layer nodes as user preference parameters. For example, the preference attribute description extracted from the hit context layer node is {"Crispness": "high", "Browning": "medium"}, indicating that the user has a high preference for crispness and a medium preference for browning in the current type of task.

[0095] S2332: Determine the physical parameter values ​​corresponding to each parameter placeholder based on user preference parameters.

[0096] Based on user preference parameters, determine the physical parameter values ​​corresponding to each parameter placeholder. Optionally, a large language model can be used to transform abstract preference attributes into physical parameters specific to each device type. For example, for the "oven baking" subtask, the user preference "Crispness: high" is mapped to specific oven parameters: increasing the temperature by 20°C and selecting convection mode. The mapping rules can be a pre-established preference-physical parameter mapping table or dynamically generated by the large language model based on device type and preference description.

[0097] Because user preference parameters are stored as device-independent abstract attributes, when a user switches to a new device of a different brand or model, it is only necessary to map the abstract attributes to the corresponding physical parameters of the new device. For example, if a user's preference for "healthy chicken breast" is "high tenderness and low fat," when the user purchases a new brand of sous-vide cooking equipment, based on this abstract attribute, the physical parameters of 65°C and 45 minutes are derived for the new device. Thus, user habits are seamlessly transferred between heterogeneous devices, and users do not need to repeatedly set up or retrain the system on the new device.

[0098] S2333: Fill the physical parameter values ​​into the corresponding parameter placeholders in the abstract control command to generate a deterministic equipment control sequence.

[0099] The determined physical parameter values ​​are filled into the corresponding parameter placeholders in the abstract control command to generate a deterministic equipment control sequence. For example, filling the abstract command bosch_oven_bake({"temperature": "[temperature]","duration": "[duration]"}) with bosch_oven_bake({"temperature": "210", "duration": "11"}) will result in a complete deterministic control command that can be directly issued.

[0100] Optionally, after obtaining the device control sequence, a virtual sandbox verification can be performed on the device control sequence to ensure that all parameters of the device control sequence are within the safety and physical constraints of the device. The method may further include: sending the device control sequence to a virtual sandbox for simulated execution; the virtual sandbox includes digital twin models of each device; when the simulation execution returns an error message, the error message is re-entered into a large language model; and using the large language model, adjusting the parameters in the device control sequence associated with the error message until the simulation execution succeeds or a preset retry limit is reached.

[0101] Device control sequences are sent to a virtual sandbox for simulated execution. The virtual sandbox contains digital twin models of each device, enabling simulation of device API calls and physical limitations.

[0102] When the simulation returns an error message, the error message is re-entered into the large language model. Based on the error message, the large language model adjusts only the parameters in the device control sequence associated with the error message, while keeping other parameters unchanged.

[0103] Repeat the above simulation execution and parameter adjustment process until the simulation execution is successful or the preset retry limit is reached (e.g., 3 times).

[0104] When the preset retry limit is reached and the simulation still fails, the device control sequence and error details are presented to the user, allowing the user to directly modify the device control sequence using natural language.

[0105] The virtual sandbox verification mechanism described above can detect and correct erroneous parameters in the control sequence before actual control of the device, thus avoiding potential security risks or execution failures that may result from directly controlling the physical device.

[0106] Finally, the control sequence generated in step S233 is sequentially sent to the corresponding target devices through a standardized device invocation protocol, controlling each target device to perform corresponding operations according to the physical parameters specified in the control sequence.

[0107] During execution, the system can monitor the execution status of each device in real time and obtain execution result feedback after the device completes its operation. After each device finishes its operation, the system can send an operation completion notification to the user, such as a voice announcement saying, "The cake is baked and ready to enjoy."

[0108] Optionally, during step S24, the multimodal sensor data acquisition and environmental status identification in step S21 can continue to be performed to achieve continuous monitoring during the cooking process. When a new risk state is detected, a new high-priority virtual interaction command can be generated to proactively initiate intervention, such as interrupting the current cooking process and prioritizing the execution of safety protection actions.

[0109] Optionally, after the target device completes the operation in step S24, feedback acquisition and memory update can also be performed. The method may further include: after the target device completes the operation, acquiring image data of the operation execution result through an image sensor; inputting the image data into a visual language model for effect evaluation and generating an evaluation result; updating the weight parameters of the hit node based on the evaluation result; and / or, updating the mapping relationship in the context layer nodes associated with the hit node based on the evaluation result.

[0110] After the target device completes its operation, image data of the operation result is acquired through an image sensor. For example, for baking tasks, the camera is controlled to capture images of the finished product from multiple angles to obtain visual images of the product.

[0111] Image data is input into a visual language model for performance evaluation, generating evaluation results. Optionally, the task objective and the acquired images are input into the VLM, which is then asked to perform multi-dimensional scoring and descriptive evaluation of the finished product, such as evaluating dimensions like color, shape, crispness, and fluidity, generating quantitative scoring results and qualitative descriptive text.

[0112] Based on the evaluation results, update the mapping relationship of the weight parameters and / or physical parameters of relevant nodes in the dynamic memory graph.

[0113] The updating of weight parameters follows a "cognitive metabolism" mechanism, which comprises three synergistic dynamic processes: Access Enhancement: After each successful task execution and positive user feedback, the weight parameter of the relevant hit node is increased, increasing its probability of being prioritized in subsequent similar scenarios. The magnitude of the weight increase is positively correlated with the satisfaction level of task execution.

[0114] Time-based decay: Weights of all nodes are periodically subject to natural decay. For preferences that haven't been accessed for a long time, their weight gradually decreases over time until it falls below the eviction threshold, at which point they are automatically archived or deleted by the system. This process simulates the natural forgetting curve of human memory, enabling the system to automatically eliminate outdated habits and prevent old preferences from occupying long-term retrieval priority.

[0115] Negative feedback penalty: When the task execution result associated with a memory node is evaluated as a failure by the visual language model (e.g., the score is significantly lower than the expected threshold), the weight parameter of that node is greatly reduced to prevent incorrect preference mapping from being repeatedly used.

[0116] Through the synergistic effect of the three dynamic processes mentioned above, the dynamic memory map has the ability to self-update and optimize. The system can achieve autonomous evolution of memory without relying on human intervention, and gradually achieve the cognitive evolution effect of "understanding the user better the more it is used".

[0117] The aforementioned cognitive metabolic mechanism, in actual operation, presents three progressively advancing and cyclical evolutionary stages, which together constitute a complete cognitive evolutionary closed loop: Phase 1: Seamless Transfer. When a user switches to a new device, the system retrieves context layer nodes from the dynamic memory graph that semantically match the new task, extracts device-independent abstract preference attributes, and maps these abstract attributes to the corresponding physical parameters of the new device based on its type and capabilities. Users do not need to repeatedly configure or retrain the system on the new device; personalized habits are seamlessly transferred between heterogeneous devices.

[0118] Phase Two: Preference Update. During daily user activity, if a user manually adjusts device parameters (e.g., changing the recommended temperature from 65°C to 68°C), the system records this action and reduces the weight of old preference nodes. Simultaneously, it creates or strengthens corresponding derivative nodes for newly observed parameter patterns. After repeated iterations, the weight of old preference nodes gradually decreases over time, while the weight of new preference nodes gradually increases due to increased access, until the system prioritizes the new preference in subsequent tasks. Thus, the system can automatically adapt to the natural shift in user habits without requiring explicit user notification of preference changes.

[0119] Phase Three: Embodied Evolutionary Loop. After each task is completed, the system collects visual data of the execution results through image sensors and submits it to the visual language model for multi-dimensional effect evaluation. Specifically, the visual language model score is compared with the expected attribute used in this task, and the deviation is calculated. If the score is consistently poor (e.g., the same attribute score is low multiple times in a row), the parameter optimizer proposes to adjust the corresponding parameters in the mapping model (e.g., adjusting the baking temperature corresponding to "Crispness: high" from +20°C to +25°C). This optimization proposal is pushed to the user for confirmation. After user confirmation, the new mapping relationship is updated in the context layer node of the dynamic memory graph, completing one evolutionary iteration. Thus, the system continuously corrects its preference mapping model based on real execution feedback from the physical world, completing the embodied evolutionary loop of "execution, feedback, evaluation, optimization, and re-execution".

[0120] The three stages described above are not a one-time process, but rather a continuous cycle that runs with user interaction: after each seamless migration, a preference update stage begins; as preference updates accumulate to a certain level, embodied evolution is triggered, and the evolved mapping relationship becomes the basis for the next seamless migration. Through this cognitive evolution loop, the system gradually transforms from a passive "device operator" into a designer and executor of personalized services, enabling the system's personalized service capabilities to continuously improve with increased usage.

[0121] In summary, the smart home control method and computer program product provided by this invention continuously collects multimodal sensor data of the home environment and identifies the environmental state. When an abnormal or risky scenario is detected, it autonomously generates virtual interaction commands with a format consistent with user interaction commands. The user interaction commands and virtual interaction commands are uniformly treated as task requests for scheduling and processing. At the same time, by obtaining the real-time function list of currently available devices, the task requests are decomposed into sub-tasks associated with device capabilities using a large language model. Based on a standardized device invocation protocol, abstract control commands with parameter placeholders are generated and mapped to deterministic device control sequences for execution. This achieves unified scheduling and proactive response to multi-source commands and transforms fuzzy natural language intentions into deterministic cross-device collaborative control sequences, effectively improving the proactive service capabilities of smart homes and the flexibility of cross-brand device collaboration. Furthermore, by combining state machines with visual language models, accurate identification and semantic understanding of environmental states are achieved; by dynamically acquiring device lists and prompt word constraints, adaptive discovery of device capabilities and task decomposition are realized; by using a standardized protocol tool list, unified abstract invocation of heterogeneous devices across brands is achieved; by using a three-layer node structure and weighted selection mechanism of a dynamic memory graph, accurate retrieval and optimal matching of historical experience are achieved; by using image acquisition and visual language model evaluation after execution, autonomous evolution based on physical world feedback is achieved; and by using digital twin simulation and automatic error correction in a virtual sandbox, reliability verification before execution is achieved. Together, these elements construct a complete closed-loop system from perception, scheduling, refinement, verification, execution to feedback evolution, which improves the system's initiative, personalization, reliability, and scalability.

[0122] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0123] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0124] If the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0126] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A smart home control method, characterized in that, include: Based on the acquired multimodal sensor data of the home environment, the current state of the home environment is identified. When the state of the home environment meets the preset triggering conditions, a virtual interaction command with the same format as the user interaction command is generated. Based on the user interaction instructions and / or the virtual interaction instructions, a task request to be executed is made. In response to the task request, based on a standardized device invocation protocol, the task request is decomposed into at least one subtask using a large language model, and each subtask is mapped to a deterministic device control sequence for the target device. The target device is controlled to perform corresponding operations according to the device control sequence.

2. The method according to claim 1, characterized in that, The step of identifying the current home environment state based on the acquired multimodal sensor data of the home environment, and generating a virtual interaction command with a format consistent with the user interaction command when the home environment state meets a preset trigger condition, includes: Continuously collect multimodal sensing data through multiple sensors deployed within the home environment; Based on the multimodal sensing data, the current home environment status is identified through a state machine; When the current home environment state meets the preset triggering conditions, obtain the semantic understanding result of the current home environment state; Based on the semantic understanding results and the preset scene action mapping rules, virtual interaction commands with the same format as user interaction commands are generated.

3. The method according to claim 1, characterized in that, In response to the task request, based on a standardized device invocation protocol, the task request is decomposed into at least one subtask using a large language model, and each subtask is mapped to a deterministic device control sequence for the target device, including: In response to the task request, a real-time list of currently available device functions is obtained, and the task request is decomposed into at least one sub-task using a large language model; wherein each sub-task is associated with a specific capability of the device. Based on a standardized device invocation protocol, an abstract control instruction with parameter placeholders is generated for each of the subtasks; The parameter placeholders in the abstract control command are mapped to specific physical parameters to be executed for the target device, thereby generating a deterministic device control sequence.

4. The method according to claim 3, characterized in that, In response to the task request, the system obtains a real-time list of currently available device functions and, using a large language model, decomposes the task request into at least one sub-task, including: Obtain the device identifier, device type, capability description, and operating status of each device currently connected to the system, and generate the real-time function list; The real-time function list and the task request are used to construct sub-tasks to extract prompt words; Based on the extracted prompt words from the subtasks, the task request is broken down into at least one subtask using a large language model; each subtask is associated with a specific capability of a device in the real-time function list.

5. The method according to claim 3, characterized in that, The method of generating abstract control instructions with parameter placeholders for each subtask based on a standardized device invocation protocol includes: Obtain a list of tools registered by each device that has connected to the standardized device invocation protocol; wherein each tool includes a tool name, function description and input parameter structure; The tool list and each of the subtasks are constructed into tool call prompts; Based on the tool call prompt words, the target tool corresponding to each subtask is matched from the tool list using the large language model; Based on the input parameter structure of the target tool, an abstract control instruction is generated containing the names of each parameter in the input parameter structure as placeholders.

6. The method according to claim 3, characterized in that, The step of mapping the parameter placeholders in the abstract control command to specific physical parameters executed for the target device, generating a deterministic device control sequence, includes: Obtain the user preference parameters corresponding to each of the subtasks; Based on the user preference parameters, determine the physical parameter values ​​corresponding to each of the parameter placeholders; The physical parameter values ​​are filled into the corresponding parameter placeholders in the abstract control command to generate a deterministic device control sequence.

7. The method according to claim 6, characterized in that, The step of obtaining the user preference parameters corresponding to each subtask includes: The task request is semantically matched with the task layer nodes in the dynamic memory graph, and / or the subtask is semantically matched with the subtask layer nodes in the dynamic memory graph to determine the hit node; the task layer node contains a task semantic vector and its associated subtask identifier, the subtask layer node contains a device type, operation template and its associated context identifier, the dynamic memory graph also includes a context layer node, the context layer node contains a preference attribute description; When there are multiple hit nodes in the semantic matching result, obtain the weight parameters of each hit node, and select the node with the highest weight parameter as the target hit node; The corresponding context layer node is determined based on the context identifier associated with the target hit node; The preference attribute description is extracted from the context layer node and used as the user preference parameter.

8. The method according to claim 7, characterized in that, The method further includes: After the target device completes the operation, image data of the operation result is acquired by an image sensor; The image data is input into a visual language model for effect evaluation, and evaluation results are generated. Based on the evaluation results, update the weight parameters of the hit node; and / or, based on the evaluation results, update the mapping relationship in the context layer nodes associated with the hit node.

9. The method according to claim 6, characterized in that, The method further includes: The device control sequence is sent to a virtual sandbox for simulated execution; the virtual sandbox includes digital twin models of each device. When the simulation execution returns an error message, the error message is re-input into the large language model; The parameters associated with the error message in the device control sequence are adjusted using the large language model until the simulation is successful or the preset retry limit is reached.

10. A computer program product, characterized in that, The computer program product includes instructions that, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1 to 9.

Images