Multi-device cooperative control method and system based on virtual-real fusion of intelligent Internet of Things and element universe

Abstract

The invention provides a multi-device cooperative control method and system based on intelligent Internet of Things and meta-universe virtual-real fusion, and relates to the technical field of meta-universe, and the method comprises the steps: receiving an operation instruction of a user in a meta-universe virtual scene, obtaining a target task parameter, determining each device role in an Internet of Things device group, and dividing master and slave devices; the control instruction is converted and executed according to a preset cooperation strategy, and finally the cooperation execution state is judged based on device feedback. According to the invention, seamless connection between the virtual scene and the real environment is realized, and the multi-device cooperation efficiency and the task execution precision are improved.

Description

Technical Field

[0001] The present invention relates to metaverse technology, and in particular to a multi-device collaborative control method and system that integrates intelligent Internet of Things and metaverse virtuality and reality. Background Art

[0002] With the rapid development of IoT technology and the Metaverse, the integration of the virtual world and the real environment has become an important trend in current technological development. As a virtual space that integrates multiple technologies, the Metaverse provides users with an immersive interactive experience, while IoT devices realize the intelligence and interconnection in the real world. At present, many studies are devoted to connecting the Metaverse with IoT devices in the real world, allowing users to control real devices through the virtual environment.

[0003] However, the existing technologies for integrating the Metaverse and the Internet of Things have some obvious shortcomings. First, most existing technologies only support the control mode of a single device and cannot achieve collaborative work between multiple devices, resulting in low efficiency in the execution of complex tasks. Secondly, the existing control methods lack a flexible role allocation mechanism and cannot dynamically adjust the functional positioning of each device according to specific task requirements, which limits the adaptability and scalability of the system. In addition, the current technology has an imperfect state feedback mechanism in the virtual-reality fusion scenario, making it difficult to synchronize the execution status of IoT devices to the Metaverse virtual environment in real time and accurately, affecting the user's control experience and decision-making efficiency.

[0004] With the growing demand for collaborative control of multiple devices in scenarios such as smart homes and industrial automation, there is an urgent need for a control method that can achieve efficient collaboration between metaverse virtual scenes and real IoT devices, so as to improve the overall collaboration capability and task execution efficiency of the system. Summary of the invention

[0005] The embodiments of the present invention provide a multi-device collaborative control method and system for the virtual-real integration of intelligent Internet of Things and Metaverse, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention, Receiving a first operation instruction of a user on a first virtual object in a metaverse virtual scene, wherein the first virtual object establishes a corresponding relationship with a first IoT device group in a real environment, and the first IoT device group includes at least two IoT devices; Acquire target task parameters of the first virtual object according to the first operation instruction, wherein the target task parameters include task type information, task execution location information, and device coordination information; Determine the device role of each IoT device in the first IoT device group according to the target task parameters, and divide the device into master devices and slave devices based on the device roles; The target task parameter is converted into a first control instruction for the master device and a second control instruction for the slave device through a preset Internet of Things communication protocol, wherein the execution timing of the first control instruction and the second control instruction is determined based on a preset collaborative strategy library; Receive first execution status information fed back by the master device, and receive second execution status information fed back by the slave device; based on the first execution status information and the second execution status information, determine the collaborative execution status of the master device and the slave device.

[0007] Acquiring the target task parameter of the first virtual object according to the first operation instruction includes: Obtaining an operation type identifier in the first operation instruction, and obtaining a task template matching the operation type identifier from a preset task mapping database according to the operation type identifier, wherein the task template includes task type information, device capability parameters, and task flow information; Acquire first position information of the first virtual object in the metaverse virtual scene, and second position information of a second virtual object interacting with the first virtual object; determine task execution position information according to the first position information and the second position information; Acquire device capability parameters of each IoT device in the first IoT device group, wherein the device capability parameters include device type information, device function information, and device status information; filter the IoT devices in the first IoT device group according to the device capability parameters to obtain a target device set that meets the device capability parameters; Constructing a directed acyclic graph according to the task process information, determining the coordination order of each IoT device in the target device set by topologically sorting the directed acyclic graph, and generating device coordination information; The task type information, the task execution location information and the device coordination information are combined to generate the target task parameters.

[0008] Constructing a directed acyclic graph according to the task flow information, determining the coordination order of each IoT device in the target device set by topologically sorting the directed acyclic graph, and generating device coordination information includes: Extracting a task execution step sequence from the task flow information, wherein the task execution step sequence includes a plurality of subtask nodes, each of the subtask nodes including execution priority information and task dependency information; Prioritize the subtask nodes according to the execution priority information to generate an initial task execution sequence; Constructing a directed acyclic graph based on the task dependency information, wherein the nodes of the directed acyclic graph are the subtask nodes, and the edges of the directed acyclic graph represent the dependency relationships between the subtask nodes; Performing topological sorting on the directed acyclic graph to generate a task execution sequence that takes dependency relationships into consideration; Obtaining current workload information and remaining resource information of each IoT device in the target device set; matching the IoT devices in the target device set with the subtask nodes in the task execution sequence; performing weighted summation of the workload information and the remaining resource information to calculate the task execution capability score of each IoT device; Sort multiple IoT devices matched to each subtask node according to the task execution capability score, and select the IoT device with the highest score as the execution device of the subtask node; Device coordination information is generated according to the execution devices corresponding to each subtask node and the task execution sequence.

[0009] The target task parameter is converted into a first control instruction for the master device and a second control instruction for the slave device through a preset Internet of Things communication protocol, wherein the execution timing of the first control instruction and the second control instruction is determined based on a preset collaborative strategy library and includes: According to the task type information of the target task parameter, the trigger condition information in the collaborative strategy template is matched to determine the target collaborative strategy template; according to the device role definition information in the target collaborative strategy template, the IoT devices in the first IoT device group are divided into master devices and slave devices; Acquire a basic instruction set corresponding to the device types of the master device and the slave device from a preset instruction template library; determine the instruction execution timing of the master device and the slave device based on the instruction timing rule in the target collaboration strategy template; and convert the basic instruction set into a first control instruction sequence and a second control instruction sequence containing specific execution timestamps according to the instruction execution timing; The first control instruction sequence and the second control instruction sequence are respectively encapsulated into a first control instruction and a second control instruction that conform to a preset Internet of Things communication protocol.

[0010] When the collaborative execution status indicates that the task is abnormal, sending a task adjustment instruction to the master device and the slave device includes: When the response delay of the master device or the slave device exceeds the preset delay threshold, or the resource usage rate of the master device or the slave device exceeds the preset resource threshold, or the difference between the first task completion degree corresponding to the master device and the second task completion degree corresponding to the slave device exceeds the preset progress threshold, the task is deemed abnormal; A task adjustment instruction is sent to the master device and the slave device, wherein the task adjustment instruction includes at least one of an instruction to suspend execution of a current task and an instruction to switch to a backup execution scheme.

[0011] The method further comprises: When the collaborative execution status indicates that the task is abnormal, a task adjustment instruction is sent to the master device and the slave device; the display status of the first virtual object in the metaverse virtual scene is updated according to the collaborative execution status to achieve status synchronization between the first virtual object and the first Internet of Things device group.

[0012] A second aspect of an embodiment of the present invention provides a multi-device collaborative control system for integrating the virtual and real aspects of intelligent IoT and Metaverse, including: A first unit is configured to receive a first operation instruction of a user on a first virtual object in a virtual scene of a metaverse, wherein the first virtual object establishes a corresponding relationship with a first IoT device group in a real environment, and the first IoT device group includes at least two IoT devices; A second unit is used to obtain target task parameters of the first virtual object according to the first operation instruction, where the target task parameters include task type information, task execution location information, and device coordination information; A third unit is used to determine a device role of each IoT device in the first IoT device group according to the target task parameters, and divide the device into a master device and a slave device based on the device role; A fourth unit is used to convert the target task parameter into a first control instruction for the master device and a second control instruction for the slave device through a preset Internet of Things communication protocol, wherein the execution timing of the first control instruction and the second control instruction is determined based on a preset collaborative strategy library; The fifth unit is used to receive the first execution status information fed back by the master device, and to receive the second execution status information fed back by the slave device; based on the first execution status information and the second execution status information, determine the collaborative execution status of the master device and the slave device.

[0013] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0014] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0015] The beneficial effects of this application are as follows: The present invention realizes the coordinated control of multiple IoT devices in the real environment by operating virtual objects in the metaverse virtual scene, realizes an intelligent interactive mode of virtual-real integration, and improves the user operation experience.

[0016] By setting target task parameters and device role division, the system can automatically determine the master-slave device relationship and generate corresponding control instructions, effectively solving the complexity of multi-device collaborative control and improving the flexibility and adaptability of the intelligent Internet of Things system.

[0017] By judging the collaborative execution status based on the execution status information fed back by the devices, real-time monitoring and dynamic adjustment of multi-device collaborative tasks can be achieved, ensuring the reliable execution of collaborative tasks and system stability, and improving the overall performance of the intelligent Internet of Things system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic flow chart of a multi-device collaborative control method for integrating the virtual and real aspects of intelligent IoT and Metaverse according to an embodiment of the present invention; Figure 2 Generate a complete flow chart for the target task parameters of the embodiment of the present invention; Figure 3 This is a flowchart of collaborative control of IoT devices according to an embodiment of the present invention; Figure 4 The figure is a complete flow chart of collaborative execution status monitoring according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0021] Figure 1 Schematic diagram of the process of multi-device collaborative control method for integrating virtual and real world with intelligent IoT and Metaverse according to an embodiment of the present invention. Figure 1As shown, the method includes: Receiving a first operation instruction of a user on a first virtual object in a metaverse virtual scene, wherein the first virtual object establishes a corresponding relationship with a first IoT device group in a real environment, and the first IoT device group includes at least two IoT devices; Acquire target task parameters of the first virtual object according to the first operation instruction, wherein the target task parameters include task type information, task execution location information, and device coordination information; Determine the device role of each IoT device in the first IoT device group according to the target task parameters, and divide the device into master devices and slave devices based on the device roles; The target task parameter is converted into a first control instruction for the master device and a second control instruction for the slave device through a preset Internet of Things communication protocol, wherein the execution timing of the first control instruction and the second control instruction is determined based on a preset collaborative strategy library; Receive first execution status information fed back by the master device, and receive second execution status information fed back by the slave device; based on the first execution status information and the second execution status information, determine the collaborative execution status of the master device and the slave device.

[0022] In an optional implementation, acquiring the target task parameter of the first virtual object according to the first operation instruction includes: Obtaining an operation type identifier in the first operation instruction, and obtaining a task template matching the operation type identifier from a preset task mapping database according to the operation type identifier, wherein the task template includes task type information, device capability parameters, and task flow information; Acquire first position information of the first virtual object in the metaverse virtual scene, and second position information of a second virtual object interacting with the first virtual object; determine task execution position information according to the first position information and the second position information; Acquire device capability parameters of each IoT device in the first IoT device group, wherein the device capability parameters include device type information, device function information, and device status information; filter the IoT devices in the first IoT device group according to the device capability parameters to obtain a target device set that meets the device capability parameters; Constructing a directed acyclic graph according to the task process information, determining the coordination order of each IoT device in the target device set by topologically sorting the directed acyclic graph, and generating device coordination information; The task type information, the task execution location information and the device coordination information are combined to generate the target task parameters.

[0023] Extract the operation type identifier from the first operation instruction. The identifier is a unique identifier of the operation type, which can indicate a specific task type, such as "data acquisition" or "equipment control". By parsing the structure of the operation instruction, the operation type identifier is extracted and stored for subsequent retrieval.

[0024] According to the extracted operation type identifier, the matching task template is retrieved from the preset task mapping database. The task template contains task type information, device capability parameters and task flow information. The task type information describes the nature of the task, such as "monitoring" or "execution"; the device capability parameters include the function and status of the device; and the task flow information defines the execution steps and sequence of the task.

[0025] Obtain the position information of the first virtual object in the virtual scene of the metaverse, and the position information of the second virtual object that interacts with it. These position information are usually expressed in the form of coordinates, such as "X: 100, Y: 200, Z: 300". By comparing the position information of these two objects, the execution position information of the task can be determined, that is, where the task will be performed.

[0026] The device capability parameters of each IoT device are obtained from the first IoT device group. The device capability parameters include device type information (such as sensor, actuator), device function information (such as data collection, control instructions), and device status information (such as online, offline). This information can help the system understand the capabilities and status of available devices.

[0027] According to the acquired device capability parameters, the devices in the first IoT device group are screened to obtain a target device set that meets the specific capability parameters. For example, if the task requires a temperature sensor, only devices with this function are selected. By analyzing the device capabilities, it can be ensured that the selected devices can meet the task requirements.

[0028] A directed acyclic graph (DAG) is constructed based on the task flow information. The graph is used to represent the various steps of task execution and their dependencies. Each node represents a task step, and the edge represents the dependency between the steps. By topologically sorting the graph, the coordination order of each IoT device in the target device set can be determined.

[0029] The topological sorting results generate device coordination information, indicating the coordination order and interaction mode of each device in task execution. This information is crucial for the smooth execution of the task, ensuring that the devices can operate in the correct order.

[0030] The task type information, task execution location information and equipment coordination information are combined to generate the target task parameters. These parameters will serve as the basis for subsequent task execution, ensuring that the system can effectively schedule and manage the operation of each device.

[0031] Assume that the first operation instruction is "start temperature monitoring" and the operation type is identified as "temperature monitoring". In the task mapping database, find the corresponding task template, assume that its task type information is "monitoring", the equipment capability parameters include "temperature sensor", and the task flow information is "start monitoring->data collection->data upload".

[0032] The first virtual object is located at coordinates (100, 200, 300), and the second virtual object is located at coordinates (150, 250, 350), so the task execution position information is (125, 225, 325), which is the midpoint between the two.

[0033] The device capability parameters obtained from the IoT device group show that there are three devices: device A (temperature sensor, online), device B (humidity sensor, online), and device C (actuator, offline). After filtering, the target device set is device A.

[0034] The constructed directed acyclic graph contains a node "start monitoring", which points to "data collection", and then to "data upload". After topological sorting, the coordination order is that device A collects data first and then uploads data.

[0035] The target task parameters finally generated include task type information "monitoring", task execution location information (125, 225, 325), and equipment coordination information to ensure that the task can be executed smoothly.

[0036] Figure 2 A complete flow chart is generated for the target task parameters of the embodiment of the present invention: This picture shows a flowchart of task operation type identification and matching task template. Virtual object information branch: From the top, the location information of the virtual object is obtained, especially the data of the first and second locations. After processing, these data are used to determine the specific location information of the task execution, providing spatial positioning basis for the execution of subsequent tasks. Task template branch: Mainly responsible for obtaining device capability parameters and screening the target device set. This branch is directly connected to the core link of the process, that is, combining and generating target task parameters. Task process information branch: First, a directed acyclic graph is constructed to determine the device coordination order, and then detailed device coordination information is generated. This information will eventually be input into the core link. The information of these three branches is finally gathered at the bottom center of the figure, which is used to combine and generate target task parameters. This final step combines the three key elements of task type, location information and device coordination to form a complete task execution plan. The entire flowchart clearly shows the complete logical chain from task identification to parameter generation, reflecting the modular design concept of the system and the hierarchical nature of information processing. The design of this process fully considers the various types of information required in the task execution process and organizes them in an orderly manner to ensure the accuracy and efficiency of task execution. Each branch performs its own duties and cooperates with each other, ultimately realizing the transformation process from task identification to the generation of specific execution parameters.

[0037] In an optional implementation, a directed acyclic graph is constructed according to the task flow information, and the coordination order of each IoT device in the target device set is determined by topologically sorting the directed acyclic graph, and generating device coordination information includes: Extracting a task execution step sequence from the task flow information, wherein the task execution step sequence includes a plurality of subtask nodes, each of the subtask nodes including execution priority information and task dependency information; Prioritize the subtask nodes according to the execution priority information to generate an initial task execution sequence; Constructing a directed acyclic graph based on the task dependency information, wherein the nodes of the directed acyclic graph are the subtask nodes, and the edges of the directed acyclic graph represent the dependency relationships between the subtask nodes; Performing topological sorting on the directed acyclic graph to generate a task execution sequence that takes dependency relationships into consideration; Obtaining current workload information and remaining resource information of each IoT device in the target device set; matching the IoT devices in the target device set with the subtask nodes in the task execution sequence; performing weighted summation of the workload information and the remaining resource information to calculate the task execution capability score of each IoT device; Sort multiple IoT devices matched to each subtask node according to the task execution capability score, and select the IoT device with the highest score as the execution device of the subtask node; Device coordination information is generated according to the execution devices corresponding to each subtask node and the task execution sequence.

[0038] Extract the task execution step sequence from the task flow information. The task flow information is usually recorded in a configuration file in JSON or XML format, which contains the execution information of multiple subtask nodes. Each subtask node contains at least the following information: task ID, task name, execution priority value, list of pre-dependent task IDs, task description, and required resource conditions. For example, the task flow information in a smart home scenario may contain subtask nodes such as "turn on the air conditioner", "adjust the temperature", and "open the curtains", among which "adjust the temperature" depends on "turn on the air conditioner" to be completed first.

[0039] By parsing the configuration file, the system extracts all subtask node information and forms an initial task list. For example, a smart manufacturing scenario contains five subtask nodes: A (assembly base, priority 3), B (install motherboard, priority 2), C (connect power, priority 2), D (install shell, priority 1), E (system test, priority 1), where the task dependencies are: B depends on A, C depends on B, D depends on A, and E depends on C and D.

[0040] Sort the subtask nodes according to the execution priority information. Priority is usually expressed as an integer, and the higher the value, the higher the priority. Sorting uses a stable sorting algorithm (such as merge sort) to ensure that tasks of the same priority maintain their original relative order. For the above example, the initial task execution sequence after sorting is: A(3), B(2), C(2), D(1), E(1).

[0041] Construct a directed acyclic graph based on task dependency information. First, create a graph data structure with each subtask node as a node in the graph and dependencies as directed edges. If task X depends on task Y, add a directed edge from Y to X, indicating that Y must be completed before X. In the above example, the constructed directed acyclic graph contains the edges: A→B, A→D, B→C, C→E, D→E.

[0042] To ensure that the constructed graph has no loops (circular dependencies), the system implements a loop detection algorithm. A common method is depth-first search, which records the access status (not visited, visiting, visited). If a loop is found, the system will prompt the existence of a circular dependency and terminate the process.

[0043] Perform topological sorting on the directed acyclic graph to generate a task execution sequence that takes into account dependencies. Topological sorting is implemented based on the Kahn algorithm: first find all nodes with in-degree 0 (tasks without dependencies) and add them to the queue; loop to take nodes out of the queue, add them to the result sequence, and reduce the in-degree of all its adjacent nodes; if the in-degree of a node becomes 0, add it to the queue; until the queue is empty. For the above example, the topological sorting results are: A, B, C, D, E or A, B, D, C, E or A, D, B, C, E, all of which satisfy the dependency relationship. When multiple nodes with in-degree 0 exist at the same time, the node with higher priority is selected first, so the final task execution sequence is: A, B, D, C, E.

[0044] Through the API interface of the IoT gateway or device management platform, the current workload information and remaining resource information of each IoT device in the target device set are obtained. The workload information includes CPU usage, memory occupancy, current task queue length, etc.; the remaining resource information includes available memory, storage space, power percentage, etc.

[0045] Based on the device capability description file, the system matches IoT devices in the target device set with subtask nodes in the task execution sequence. Each subtask node has execution condition requirements, such as processor type, minimum memory requirements, specific sensors, etc. The system compares these requirements with device capabilities and selects a subset of devices that meet the conditions. For example, the "adjust temperature" task in a smart home scenario may require devices such as smart thermostats and smart air conditioners.

[0046] Calculate the task execution capability score of each IoT device. First, define the workload weight W1 and the remaining resource weight W2 (for example, W1=0.4, W2=0.6). For the workload L, the value range is 0-100%, the lower the better, so calculate (100-L) as a positive indicator. For the remaining resources R, the value range is 0-100%, the higher the better. The device score S is calculated as: S = W1×(100-L) + W2×R.

[0047] The current workload of device D1 is 30%, and the remaining resources are 70%. Its score is: 0.4×(100-30) +0.6×70 = 28 + 42 = 70 points. The current workload of device D2 is 10%, and the remaining resources are 60%. Its score is: 0.4×(100-10) + 0.6×60 = 36 + 36 = 72 points.

[0048] Sort the multiple IoT devices that match each subtask node, and select the device with the highest score as the execution device of the subtask node. If the scores are the same, further consider factors such as device response time and historical execution success rate. In the above example, the task will be assigned to device D2 with a score of 72.

[0049] Generate device coordination information based on the execution devices and task execution sequence corresponding to each subtask node. Device coordination information is stored in JSON format, including task ID, execution device ID, execution sequence, expected start time, latest completion time, communication protocol information, etc.

[0050] The device coordination information is sent to the task scheduling execution module, which is responsible for sending control instructions to IoT devices according to the order and schedule in the coordination information and monitoring the task execution status. If a task fails, the system will recalculate the device score and select an alternative device, or suspend the execution of related dependent tasks.

[0051] Through the above steps, the system realizes the intelligent collaboration of IoT devices based on task process information, ensuring that tasks are efficiently executed in the correct dependency order, while taking into account device load balancing to improve the overall system operation efficiency.

[0052] In an optional implementation, the target task parameter is converted into a first control instruction for the master device and a second control instruction for the slave device through a preset Internet of Things communication protocol, wherein the execution timing of the first control instruction and the second control instruction is determined based on a preset collaborative strategy library and includes: According to the task type information of the target task parameter, the trigger condition information in the collaborative strategy template is matched to determine the target collaborative strategy template; according to the device role definition information in the target collaborative strategy template, the IoT devices in the first IoT device group are divided into master devices and slave devices; Acquire a basic instruction set corresponding to the device types of the master device and the slave device from a preset instruction template library; determine the instruction execution timing of the master device and the slave device based on the instruction timing rule in the target collaboration strategy template; and convert the basic instruction set into a first control instruction sequence and a second control instruction sequence containing specific execution timestamps according to the instruction execution timing; The first control instruction sequence and the second control instruction sequence are respectively encapsulated into a first control instruction and a second control instruction that conform to a preset Internet of Things communication protocol.

[0053] The preset collaborative strategy library contains multiple collaborative strategy templates to guide the collaborative work between devices. Each collaborative strategy template includes trigger condition information, device role definition information, command timing rules and task monitoring rules. For example, in the home scene, there is a "smart lighting" collaborative strategy template, whose trigger condition is "the user enters the room and the light is insufficient"; the device role is defined as "the main light is the main device, the auxiliary light and the curtain are the slave devices"; the command timing rule is "the main light is turned on first, and after the brightness reaches 30%, the auxiliary light is turned on, and finally the curtain is closed"; the task monitoring rule is "detect the activity of people in the room, and automatically turn off if there is no activity for 30 minutes".

[0054] The target collaborative strategy template is determined by matching the trigger condition information in the collaborative strategy template with the task type information of the target task parameter. Specifically, when the target task parameter is "Conference Room Lighting Control, Task Type is 'Meeting Start'", the system searches for a template whose trigger condition includes "Meeting Start" in the collaborative strategy library. If the "Conference Room Intelligent Lighting" template is found, it is determined as the target collaborative strategy template.

[0055] According to the device role definition information in the target collaboration strategy template, the IoT devices in the first IoT device group are divided into master devices and slave devices. For example, in the "Conference Room Intelligent Lighting" template, the device role is defined as "central lighting as master device, side lighting and projection screen as slave devices". When the first IoT device group includes "central lighting A001", "side spotlight B002" and "electric projection screen C003" in the conference room, according to the role definition, it is determined that "central lighting A001" is the master device, and "side spotlight B002" and "electric projection screen C003" are slave devices.

[0056] Get the basic instruction set corresponding to the device type of the master device and the slave device from the preset instruction template library. The instruction template library stores basic operation instructions for different types of devices. For example, the basic instruction set of lighting equipment includes {open (position, brightness), close (position), adjust brightness (position, brightness value)}, and the basic instruction set of projection screen includes {expand (position, expansion degree), fold (position, fold degree)}. For the master device "central lighting A001", get the basic instruction set of lighting equipment; for the slave devices "side spotlight B002" and "electric projection screen C003", get the basic instruction sets of lighting equipment and projection screen respectively.

[0057] According to the task execution location information in the target task parameters, the location parameters in the basic instruction set are updated. When the target task parameters include "Conference room location: 301 conference room, area B, 3rd floor", the location parameters in the lighting equipment basic instruction set are updated to "301 conference room, area B, 3rd floor", and the specific instructions become {Open (301 conference room, area B, 3rd floor, brightness), Close (301 conference room, area B, 3rd floor), Adjust brightness (301 conference room, area B, 3rd floor, brightness value)}.

[0058] Based on the instruction timing rules in the target collaboration strategy template, determine the instruction execution timing of the master device and the slave device. For example, the instruction timing rule in the "Conference Room Intelligent Lighting" template is "the main light is turned on to 60% brightness first, and the side spotlights are turned on to 40% brightness after 2 seconds, and the projection screen is unfolded to 100% at the same time." According to this rule, it is determined that the master device "central lighting A001" is executed first, and the slave devices "side spotlights B002" and "electric projection screen C003" are executed later, and the execution of "side spotlights B002" needs to wait for 2 seconds after the execution of the master device is completed.

[0059] According to the instruction execution sequence, the basic instruction set is converted into a first control instruction sequence and a second control instruction sequence containing a specific execution timestamp. Assuming that the current time is "2025-04-20 14:30:00", the first control instruction sequence generated is {Open (3rd Floor, Area B, 301 Conference Room, 60%), 2025-04-20 14:30:00}, and the second control instruction sequence is {Open (3rd Floor, Area B, 301 Conference Room, 40%), 2025-04-20 14:30:02} and {Expand (3rd Floor, Area B, 301 Conference Room, 100%), 2025-04-20 14:30:02}.

[0060] The first control instruction sequence and the second control instruction sequence are respectively encapsulated into a first control instruction and a second control instruction that conform to a preset Internet of Things communication protocol.

[0061] Figure 3 This is a flowchart of collaborative control of IoT devices according to an embodiment of the present invention: This picture shows a detailed collaborative control process of IoT devices. In actual applications, appropriate collaborative strategy templates are matched according to the task type information in the target task parameters, and IoT devices are classified into two categories: master devices and slave devices according to the device role definition information in these templates. Subsequently, the system extracts the basic instruction sets corresponding to the master and slave device types from the preset instruction template library, and updates the position parameters in these instruction sets according to the task execution location information. After determining the instruction execution timing, the system will clarify the specific instruction execution sequence of the master and slave devices, including the execution sequence, trigger conditions, and waiting time. Finally, the system converts these basic instruction sets into the first control instruction sequence and the second control instruction sequence containing specific execution timestamps. These control instruction sequences will be encapsulated as control instructions that comply with the preset IoT communication protocol, which contain key elements such as instruction identification information, device identification information, instruction content information, and verification information. This complete process ensures effective collaboration and precise control between IoT devices.

[0062] In the existing technology, IoT device control usually adopts a centralized control mode, where a single controller sends independent commands to each device, lacking the ability to coordinate between devices. For example, in traditional smart home systems, operations such as turning on lighting, adjusting air conditioning, and controlling curtains require users to issue commands separately, and devices cannot automatically work together according to scene requirements.

[0063] In addition, the instruction conversion process in the prior art is usually based on a one-to-one mapping of fixed logic, lacking precise control of the timing relationship between devices. When multiple devices work together, they often rely on the polling of the controller or the autonomous perception of the device, making it difficult to achieve precise timing control, resulting in poor coordination effects.

[0064] The starting point of this application improvement is to realize the intelligent collaborative control of IoT devices, introduce collaborative strategy library and instruction template library, and intelligently convert abstract task parameters into specific control instructions with precise execution timing. Through the division of device roles and the definition of timing rules, the master-slave collaboration mode between devices is realized, so that complex tasks can be completed through the collaboration of multiple devices.

[0065] The improved technology has significant effects: first, it realizes intelligent collaboration among IoT devices, and a single task instruction can trigger multiple devices to work together according to preset rules; second, it accurately controls the execution timing of instructions to ensure the smoothness of collaborative operations; third, it realizes the standardization and reuse of scenarios through collaborative strategy templates, reducing the complexity of system configuration; finally, instruction encapsulation based on standard communication protocols improves the compatibility and scalability of the system.

[0066] In an optional implementation, when the collaborative execution status indicates that the task is abnormal, sending the task adjustment instruction to the master device and the slave device includes: When the response delay of the master device or the slave device exceeds the preset delay threshold, or the resource usage rate of the master device or the slave device exceeds the preset resource threshold, or the difference between the first task completion degree corresponding to the master device and the second task completion degree corresponding to the slave device exceeds the preset progress threshold, the task is deemed abnormal; A task adjustment instruction is sent to the master device and the slave device, wherein the task adjustment instruction includes at least one of an instruction to suspend execution of a current task and an instruction to switch to a backup execution scheme.

[0067] During the execution of collaborative tasks, the system will monitor the execution status of the master device and the slave device in real time, and determine whether there is a task abnormality based on the preset threshold. The specific judgment mechanism is as follows: Heartbeat detection signals are sent to the master and slave devices regularly, and the response time of the devices is recorded. When the response delay of a device is detected to exceed the preset delay threshold, the system will determine that the task execution is abnormal.

[0068] Set the preset delay threshold to 200 milliseconds, and the system sends a heartbeat detection signal to the master device every 5 seconds. Under normal circumstances, the master device should respond within 50-100 milliseconds. If the master device response time reaches 250 milliseconds in a certain detection, exceeding the preset threshold of 200 milliseconds, the system determines that the master device may have problems such as network delay or insufficient processing capacity, triggering a task exception.

[0069] Obtain resource usage information such as CPU usage, memory usage, network bandwidth usage, etc. of the master and slave devices in real time. When the key resource usage of any device exceeds the preset resource threshold, the system will determine that the task execution is abnormal.

[0070] Assume that the system sets the CPU usage threshold to 85% and the memory usage threshold to 80%. During the task execution, if the CPU usage of the slave device is detected to reach 92%, exceeding the preset threshold of 85%, the system will determine that the slave device may face a resource bottleneck, which may lead to a decrease in task execution efficiency or task failure, thus triggering a task exception.

[0071] For different types of tasks, the system may adjust the resource usage threshold settings. For example, for computationally intensive tasks, the CPU usage threshold may be set higher (such as 90%), while for real-time interactive tasks, it may be set lower (such as 70%).

[0072] In a collaborative task, the master and slave devices are usually responsible for different but interrelated subtasks. The system will periodically check the task completion of the two and calculate the difference. When the completion difference exceeds the preset progress threshold, the system will determine that the collaborative task execution is not synchronized and there is an abnormality.

[0073] For example, in a data processing task that requires the master and slave devices to complete synchronously, the preset progress threshold is set to 15%. If at a certain checkpoint, the master device is 75% complete, while the slave device is only 25%, the difference between the two is 50%, far exceeding the preset threshold of 15%. The system will determine that the task execution is abnormal, which may be caused by insufficient processing capacity of the slave device or unreasonable task allocation.

[0074] The calculation method of task completion may vary depending on the task type. For example, for data processing tasks, it can be calculated based on the ratio of processed data volume to total data volume; for file transfer tasks, it can be calculated based on the ratio of transferred bytes to total bytes.

[0075] When a task anomaly is detected, the system will automatically trigger the task adjustment process and send task adjustment instructions to the master and slave devices. Task adjustment instructions mainly include the following two categories: When a recoverable exception is detected, a pause instruction will be sent to the relevant device, requiring the device to temporarily stop the execution of the current task and save the current execution status.

[0076] After receiving the pause command, the device needs to save the current operation within a predetermined time (such as 500 milliseconds) and return a confirmation message. If the device cannot complete the pause operation within the predetermined time, the system will take forced pause measures.

[0077] After the pause, the system will periodically check whether the cause of the exception has been resolved. For example, if the exception is caused by excessive resource usage, the system will wait until the resource usage drops to a safe level (such as CPU usage drops below 70%) before sending a command to the device to resume execution.

[0078] When the detected anomaly is serious or the problem cannot be solved after the pause, an instruction to switch to an alternative execution plan will be sent to the device. The alternative execution plan may include replacing the execution device, adjusting the task allocation ratio, reducing the task execution accuracy requirements, etc.

[0079] When switching execution plans, the system will give priority to the plan with the least impact. For example, if only one device has tight resources, it may choose to add an auxiliary device to share the task; if the complexity of the task itself exceeds expectations, it may choose to reduce the accuracy requirements or extend the execution time limit.

[0080] When processing a large number of high-definition images, the system detected that the memory usage of the slave device reached 87% (exceeding the preset threshold of 80%), and the response delay increased from the normal 85 milliseconds to 230 milliseconds (exceeding the preset threshold of 200 milliseconds). At the same time, the master device had completed 75% of the task, while the slave device had only completed 40%, a difference of 35% exceeding the preset progress threshold of 20%.

[0081] A pause command is sent to the master and slave devices, and both devices complete the current operation, save, and confirm the pause within 300 milliseconds. System analysis found that insufficient resources on the slave device were the main problem, so it decided to switch to the pre-configured backup plan C. The system sends a switching plan instruction to all relevant devices, indicating that: the master device continues to perform the original task, but reduces the image processing accuracy from high-precision mode to standard-precision mode; the slave device task load is reduced by 40%, and is only responsible for result integration; the backup device DEVICE003 is enabled to undertake the image classification task originally responsible for the slave device; the total task execution time limit is extended from the original 30 minutes to 45 minutes; after each device confirms that it has received the switching plan instruction, it resumes task execution according to the new plan. The system continues to monitor the task execution status and confirms that the resource utilization rate and response delay of all devices have returned to the normal range, and the task completion difference is also controlled within 15%.

[0082] Through the above adjustment mechanism, the system successfully coped with abnormal situations during the task execution, avoided task failure, and ensured the final completion of the task. Although the accuracy was slightly reduced and the time consumption was increased compared with the original plan, the overall system reliability requirements were met.

[0083] Figure 4 This is a complete flow chart of collaborative execution status monitoring according to an embodiment of the present invention: This picture shows a complete flow chart of collaborative execution status monitoring. Starting from the top level, three key parameters are monitored simultaneously: response latency, resource utilization, and task completion difference. When any of these parameters exceeds its preset threshold, the system will determine the task status as abnormal. Specifically, the system will check whether the response latency exceeds the preset latency threshold, monitor whether the resource utilization exceeds the preset resource threshold, and track in real time whether the task completion difference exceeds the preset progress threshold. Once the system confirms that the task is abnormal, two parallel response measures will be triggered immediately: sending a command to suspend the current task execution to the master device, and sending a command to switch to the backup execution plan to the slave device. This design ensures that the system can respond quickly and take appropriate remedial measures when an abnormal situation occurs, reflecting the fault tolerance and reliability of the system. The entire monitoring process forms a closed-loop management mechanism that can effectively ensure the stable execution of tasks and the rational use of resources.

[0084] In an optional implementation, the method further includes: When the collaborative execution status indicates that the task is abnormal, a task adjustment instruction is sent to the master device and the slave device; the display status of the first virtual object in the metaverse virtual scene is updated according to the collaborative execution status to achieve status synchronization between the first virtual object and the first Internet of Things device group.

[0085] In the process of collaborative execution of tasks between the Metaverse virtual scene and IoT devices, when the collaborative execution status indicates that the task is abnormal, the system needs to handle the exception in a timely manner and keep the status of the virtual object and the physical device synchronized. The collaborative execution status monitoring module monitors the execution status of the master device and the slave device in real time. The collaborative execution status can be judged by regularly collecting device feedback data, including but not limited to device operating parameters, task completion, abnormal alarm information, etc. When the monitoring module detects that the collaborative execution status indicates that the task is abnormal, the system will immediately trigger the exception handling process.

[0086] Device execution parameters exceed preset thresholds: for example, when the temperature of the master device exceeds 75°C or the response time of the slave device exceeds 200 milliseconds; abnormal task progress: such as the task execution progress deviates from expectations by more than 20% or there is no progress update for 5 consecutive minutes; abnormal communication: such as the communication interruption between devices exceeds 30 seconds or the data transmission error rate exceeds 5%; resource exhaustion: such as the device power is less than 15% or the storage space is less than 10MB; security alert: such as unauthorized access attempts or abnormal operation instructions are detected.

[0087] Generate corresponding task adjustment instructions based on the exception type. These instructions can be preset exception response strategies or dynamically generated optimization solutions. Task adjustment instructions are sent to the master and slave devices through a secure communication channel to instruct the devices to perform corresponding adjustment measures.

[0088] Command identifier: a unique identification code, such as "TA20231205001"; Command priority: identifies the urgency of processing, ranging from 1 to 5, with 5 being the highest priority; Command type: such as "parameter adjustment", "task pause", "task restart", "resource reallocation", etc.; Command parameters: provide specific parameters based on the command type, such as temperature adjustment target value, reallocated resource quota, backup communication channel, etc.; Execution time limit: how long the command needs to be executed, such as "immediately" or "within 30 seconds"; Feedback requirements: specify the information content and method that the device needs to feedback.

[0089] Ensure accurate transmission of instructions through reliable communication protocols, such as using the MQTT protocol to send instructions and requiring the device to return confirmation information of instruction reception. If no confirmation is received within the preset time (such as 5 seconds), the system will automatically resend the instruction, and the maximum number of retries is 3. If no confirmation is received after 3 retries, the system will record the communication abnormality and upgrade the abnormality level.

[0090] When a task exception is detected and processed, the system needs to promptly update the display status of the corresponding virtual objects in the metaverse virtual scene to reflect the actual status of the physical device and achieve virtual-real synchronization.

[0091] Basic display properties: shape, size, color, position, transparency, etc.; dynamic display properties: action status, special effects, particle effects, etc.; interactive properties: selectable status, interactive prompts, etc.; status identification: running indicator light, status icon, text description, etc.

[0092] The updated display status is calculated based on the latest collaborative execution status information. For example, for a master device that detects an overtemperature, the system performs the following display status updates: Change the color of virtual objects: from blue during normal operation (RGB: 0,120,255) to red for abnormal warning (RGB: 255,60,60); Add visual effects: Add heat wave texture effect around virtual objects, with transparency set to 60% and fluctuation frequency of 2Hz; Display status icon: Display temperature warning icon above virtual objects, with a size of 25% of the height of virtual objects; Modify interaction prompt: When the user hovers the cursor over the virtual object, it will display "Device temperature is abnormal (78℃), cooling process is being executed"; Adjust motion state: Reduce the motion speed of virtual objects by 30% to simulate the state where the actual device reduces performance due to cooling needs.

[0093] This update instruction is sent to the Metaverse scene rendering engine, which is responsible for applying the update to the virtual scene. After receiving the instruction, the rendering engine will perform the update operation according to the specified priority and transition method to ensure that the display state of the virtual object changes smoothly and naturally, while accurately reflecting the actual state of the physical device.

[0094] In addition, to ensure that users are informed of device status changes in a timely manner, the system will also display status change notifications on the user interface. For example, a pop-up notification will appear in the upper right corner of the interface: "Device temperature is abnormal, cooling is being performed, and it is expected to return to normal within 60 seconds." The notification will be displayed for 10 seconds, and users can click on the notification to view more detailed abnormality handling information.

[0095] Through the above mechanism, the system realizes real-time synchronization of the status of the first virtual object and the first IoT device group, allowing users to intuitively understand the working status of the physical device and intervene or operate through the virtual interface when necessary. At the same time, this virtual-real synchronization mechanism also facilitates subsequent remote maintenance, fault diagnosis and performance optimization.

[0096] A second aspect of an embodiment of the present invention provides a multi-device collaborative control system for integrating the virtual and real aspects of intelligent IoT and Metaverse, including: A first unit is configured to receive a first operation instruction of a user on a first virtual object in a virtual scene of a metaverse, wherein the first virtual object establishes a corresponding relationship with a first IoT device group in a real environment, and the first IoT device group includes at least two IoT devices; A second unit is used to obtain target task parameters of the first virtual object according to the first operation instruction, where the target task parameters include task type information, task execution location information, and device coordination information; A third unit is used to determine a device role of each IoT device in the first IoT device group according to the target task parameters, and divide the device into a master device and a slave device based on the device role; A fourth unit is used to convert the target task parameter into a first control instruction for the master device and a second control instruction for the slave device through a preset Internet of Things communication protocol, wherein the execution timing of the first control instruction and the second control instruction is determined based on a preset collaborative strategy library; The fifth unit is used to receive the first execution status information fed back by the master device, and to receive the second execution status information fed back by the slave device; based on the first execution status information and the second execution status information, determine the collaborative execution status of the master device and the slave device.

[0097] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0098] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0099] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-device collaborative control method for the integration of intelligent IoT and metaverse virtuality and reality, characterized in that: include: Receiving a first operation instruction of a user on a first virtual object in a metaverse virtual scene, wherein the first virtual object establishes a corresponding relationship with a first IoT device group in a real environment, and the first IoT device group includes at least two IoT devices; Acquire target task parameters of the first virtual object according to the first operation instruction, wherein the target task parameters include task type information, task execution location information, and device coordination information; Determine the device role of each IoT device in the first IoT device group according to the target task parameters, and divide the device into master devices and slave devices based on the device roles; The target task parameter is converted into a first control instruction for the master device and a second control instruction for the slave device through a preset Internet of Things communication protocol, wherein the execution timing of the first control instruction and the second control instruction is determined based on a preset collaborative strategy library; receiving first execution status information fed back by the master device, and receiving second execution status information fed back by the slave device; Based on the first execution status information and the second execution status information, a collaborative execution status of the master device and the slave device is determined.

2. The method according to claim 1, characterized in that Acquiring the target task parameter of the first virtual object according to the first operation instruction includes: Obtaining an operation type identifier in the first operation instruction, and obtaining a task template matching the operation type identifier from a preset task mapping database according to the operation type identifier, wherein the task template includes task type information, device capability parameters, and task flow information; Acquire first position information of the first virtual object in the metaverse virtual scene, and second position information of a second virtual object interacting with the first virtual object; determine task execution position information according to the first position information and the second position information; Acquire device capability parameters of each IoT device in the first IoT device group, wherein the device capability parameters include device type information, device function information, and device status information; filter the IoT devices in the first IoT device group according to the device capability parameters to obtain a target device set that meets the device capability parameters; Constructing a directed acyclic graph according to the task process information, determining the coordination order of each IoT device in the target device set by topologically sorting the directed acyclic graph, and generating device coordination information; The task type information, the task execution location information and the device coordination information are combined to generate the target task parameters.

3. The method according to claim 2, characterized in that Constructing a directed acyclic graph according to the task flow information, determining the coordination order of each IoT device in the target device set by topologically sorting the directed acyclic graph, and generating device coordination information includes: Extracting a task execution step sequence from the task flow information, wherein the task execution step sequence includes a plurality of subtask nodes, each of the subtask nodes including execution priority information and task dependency information; Prioritize the subtask nodes according to the execution priority information to generate an initial task execution sequence; Constructing a directed acyclic graph based on the task dependency information, wherein the nodes of the directed acyclic graph are the subtask nodes, and the edges of the directed acyclic graph represent the dependency relationships between the subtask nodes; Performing topological sorting on the directed acyclic graph to generate a task execution sequence that takes dependency relationships into consideration; Obtaining current workload information and remaining resource information of each IoT device in the target device set; matching the IoT devices in the target device set with the subtask nodes in the task execution sequence; performing weighted summation of the workload information and the remaining resource information to calculate the task execution capability score of each IoT device; Sort multiple IoT devices matched to each subtask node according to the task execution capability score, and select the IoT device with the highest score as the execution device of the subtask node; Device coordination information is generated according to the execution devices corresponding to each subtask node and the task execution sequence.

4. The method according to claim 1, characterized in that: The target task parameter is converted into a first control instruction for the master device and a second control instruction for the slave device through a preset Internet of Things communication protocol, wherein the execution timing of the first control instruction and the second control instruction is determined based on a preset collaborative strategy library and includes: According to the task type information of the target task parameter, the trigger condition information in the collaborative strategy template is matched to determine the target collaborative strategy template; according to the device role definition information in the target collaborative strategy template, the IoT devices in the first IoT device group are divided into master devices and slave devices; Acquire a basic instruction set corresponding to the device types of the master device and the slave device from a preset instruction template library; determine the instruction execution timing of the master device and the slave device based on the instruction timing rule in the target coordination strategy template; and convert the basic instruction set into a first control instruction sequence and a second control instruction sequence containing specific execution timestamps according to the instruction execution timing; The first control instruction sequence and the second control instruction sequence are respectively encapsulated into a first control instruction and a second control instruction that conform to a preset Internet of Things communication protocol.

5. The method according to claim 1, characterized in that When the collaborative execution status indicates that the task is abnormal, sending a task adjustment instruction to the master device and the slave device includes: When the response delay of the master device or the slave device exceeds the preset delay threshold, or the resource usage rate of the master device or the slave device exceeds the preset resource threshold, or the difference between the first task completion degree corresponding to the master device and the second task completion degree corresponding to the slave device exceeds the preset progress threshold, the task is deemed abnormal; A task adjustment instruction is sent to the master device and the slave device, wherein the task adjustment instruction includes at least one of an instruction to suspend execution of a current task and an instruction to switch to a backup execution scheme.

6. The method according to claim 1, characterized in that The method further comprises: When the collaborative execution status indicates that the task is abnormal, a task adjustment instruction is sent to the master device and the slave device; the display status of the first virtual object in the metaverse virtual scene is updated according to the collaborative execution status to achieve status synchronization between the first virtual object and the first Internet of Things device group.

7. A multi-device collaborative control system integrating intelligent Internet of Things and metaverse virtuality and reality, used to implement the method as described in any one of claims 1 to 6, characterized in that: include: A first unit is configured to receive a first operation instruction of a user on a first virtual object in a virtual scene of a metaverse, wherein the first virtual object establishes a corresponding relationship with a first IoT device group in a real environment, and the first IoT device group includes at least two IoT devices; A second unit is used to obtain target task parameters of the first virtual object according to the first operation instruction, where the target task parameters include task type information, task execution location information, and device coordination information; A third unit is used to determine a device role of each IoT device in the first IoT device group according to the target task parameters, and divide the device into a master device and a slave device based on the device role; A fourth unit is used to convert the target task parameter into a first control instruction for the master device and a second control instruction for the slave device through a preset Internet of Things communication protocol, wherein the execution timing of the first control instruction and the second control instruction is determined based on a preset collaborative strategy library; A fifth unit is configured to receive the first execution status information fed back by the master device, and receive the second execution status information fed back by the slave device; Based on the first execution status information and the second execution status information, a collaborative execution status of the master device and the slave device is determined.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images