Internet of Things equipment dynamic collaboration method and device, equipment, storage medium and product

Abstract

The invention discloses an Internet of Things equipment dynamic collaboration method and device, equipment, a storage medium and a product. The method comprises the following steps: acquiring a current environment state of target equipment and an actual action of collaboration equipment; predicting a predicted action of the collaborative equipment under each candidate strategy in a strategy knowledge base based on the current environment state, and selecting a target strategy in the strategy knowledge base based on the predicted action and the actual action; and determining a current action of the target equipment based on the target strategy and the current environment state, and controlling the target equipment to execute the current action. According to the method, the target strategy is dynamically selected for the target equipment in the strategy knowledge base on the basis of the current observation state of the target equipment and the actual action of the cooperative equipment, and the target equipment is controlled based on the selected target strategy, so that the control strategy is dynamically selected for the target equipment; the problem that the control effect is poor due to the fact that a fixed strategy cannot adapt to a complex and changeable environment is solved.

Description

Technical Field

[0001] This application relates to the field of Internet of Things (IoT) technology, and in particular to a method, apparatus, device, storage medium, and product for dynamic coordination of IoT devices. Background Technology

[0002] In recent years, IoT technology and applications have continued to flourish and have been widely applied in dynamic collaborative scenarios within the Industrial Internet. In these scenarios, collaborative IoT devices typically operate using static strategies. However, when unexpected situations arise in collaborative tasks, static strategies often fail to respond and adjust in a timely manner, leading to low collaborative efficiency and difficulty in meeting the complex and ever-changing demands of industrial production. For example, if a critical piece of equipment on the production line suddenly malfunctions, other collaborative IoT devices operating according to static strategies may not be able to quickly adjust their operating modes and will continue production as planned. This not only wastes resources but may also cause the entire production process to stall, affecting product delivery time and quality.

[0003] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide a method, apparatus, device, storage medium and product for dynamic coordination of Internet of Things (IoT) devices, addressing the shortcomings of existing technologies.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a method for dynamic collaboration of IoT devices, wherein the method specifically includes: Obtain the current environmental status of the target device and the actual actions of the collaborating devices; Based on the current environmental state, predict the predicted action of the collaborative device under each candidate strategy in the strategy knowledge base, and select the target strategy in the strategy knowledge base based on the predicted action and the actual action. Based on the target strategy and the current environmental state, the current action of the target device is determined, and the target device is controlled to execute the current action.

[0006] The aforementioned dynamic coordination method for IoT devices includes candidate strategies comprising several latent variables, which are constructed based on trajectory data. Each trajectory point in the trajectory data includes environmental state and action.

[0007] The aforementioned method for dynamic collaboration of IoT devices, wherein predicting the predicted action of the collaborative device under each candidate policy in the policy knowledge base based on the current environmental state specifically includes: For each latent variable in the candidate strategy, a first conditional distribution probability of the predicted action corresponding to the latent variable is determined based on the current environmental state. The predicted action with the highest probability in the first conditional distribution is selected to obtain the predicted action of the collaborative device under the candidate strategy.

[0008] The aforementioned dynamic coordination method for IoT devices, wherein selecting a target strategy from the strategy knowledge base based on the predicted action and the actual action specifically includes: Obtain the loss difference between the predicted action and the actual action under each candidate strategy; The confidence weights of candidate strategies are determined based on all loss differences, and the target strategy is selected from the strategy knowledge base according to the confidence weights.

[0009] The aforementioned dynamic coordination method for IoT devices, wherein obtaining the loss difference between the predicted action and the actual action under each candidate strategy specifically includes: Read the latent variables corresponding to the predicted action under each candidate strategy in the strategy knowledge base; Based on the current environmental state, determine the second conditional distribution probability of the actual action under each latent variable read; The loss difference between the predicted action and the actual action under each candidate policy is determined based on the second conditional distribution probability under each latent variable.

[0010] The aforementioned dynamic coordination method for IoT devices, wherein determining the confidence weights for candidate strategies based on all loss differences specifically includes: The current confidence weight of each candidate policy in the policy knowledge base is updated exponentially based on all loss differences to obtain the updated confidence weight. The updated confidence weights are smoothed based on the smoothing parameters corresponding to the candidate strategies to obtain the confidence weights of the candidate strategies.

[0011] In the aforementioned dynamic coordination method for IoT devices, each candidate strategy uses the same smoothing parameters in the smoothing process.

[0012] The aforementioned IoT device dynamic coordination method, wherein determining the current action of the target device based on the target strategy and the current environmental state specifically includes: Determine the initial action distribution of the target device based on the current environmental state; Obtain the policy preference corresponding to the target policy, and adjust the initial action distribution based on the policy preference to obtain the target action distribution; The current action of the target device is determined based on the target action distribution.

[0013] The aforementioned method for dynamic coordination of IoT devices, wherein obtaining the strategy preference corresponding to the target strategy specifically includes: Obtain the policy category of the target policy; Based on the strategy category, select strategy preferences for the target strategy.

[0014] The aforementioned dynamic collaboration method for IoT devices, wherein the construction process of the strategy knowledge base specifically includes: Acquire trajectory data, wherein each trajectory point in the trajectory data includes environmental state and action; The latent variable distribution parameters of the trajectory points in the trajectory data are obtained by a variational autoencoder. The latent variable set is determined based on the latent variable distribution parameters, and the latent variable set is clustered into several latent variable clusters; Each cluster of latent variables is treated as a candidate policy, resulting in a policy knowledge base.

[0015] The aforementioned method for dynamic coordination of IoT devices, wherein determining the latent variable set based on the latent variable distribution parameters specifically includes: The distribution parameters of the latent variables corresponding to each trajectory point are reparameterized to obtain the latent variables corresponding to each trajectory point; The set of latent variables corresponding to all trajectory points is called the latent variable set.

[0016] The aforementioned method for dynamic coordination of IoT devices, wherein clustering the latent variables into several latent variable clusters specifically includes: The number of clusters is determined based on the silhouette coefficient, which is determined based on the average intra-cluster distance and the nearest neighbor cluster distance. The latent variable set is clustered based on the number of clusters to obtain several latent variable clusters.

[0017] The aforementioned method for dynamic coordination of IoT devices, wherein after treating each cluster of latent variables as a candidate strategy to obtain several candidate strategies, the method further includes: Obtain the policy anchor point for each candidate policy; The policy anchor is obtained to determine the policy category of the candidate policy.

[0018] The aforementioned method for dynamic collaboration of IoT devices further includes: Construct new trajectory points based on the current environment state and the current action; The strategy knowledge base is updated based on the new trajectory points.

[0019] The aforementioned dynamic collaboration method for IoT devices, wherein updating the strategy knowledge base based on the new trajectory points specifically includes: Obtain the new latent variables corresponding to the new trajectory points, and determine the candidate strategies to which the new latent variables belong; The policy anchor of the candidate policy is updated based on the new latent variables to update the policy knowledge base.

[0020] The aforementioned IoT device dynamic coordination method, wherein the update process of updating the policy anchor point of the candidate policy based on the new latent variable is as follows: , , in, This indicates the updated policy anchor. This indicates the policy anchor point before the update. Indicates a new latent variable. Indicates the learning rate. This indicates the number of latent variables contained in the latent variable cluster before the addition of the new latent variable.

[0021] In the aforementioned method for dynamic collaboration of IoT devices, the construction and updating processes of the strategy knowledge base are executed by cloud devices.

[0022] A second aspect of this application provides a dynamic coordination device for Internet of Things (IoT) devices, wherein the dynamic coordination device for IoT devices specifically includes: The acquisition module is used to acquire the current environmental status of the target device and the actual actions of the collaborating devices. The selection module is used to predict the predicted action of the collaborative device under each candidate strategy in the strategy knowledge base based on the current environmental state, and to select the target strategy in the strategy knowledge base based on the predicted action and the actual action. The control module is used to determine the current action of the target device based on the target strategy and the current environmental state, and to control the target device to execute the current action.

[0023] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the IoT device dynamic collaboration method as described above.

[0024] A fourth aspect of this application provides an electronic device comprising: a processor and a memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in any of the above-described methods for dynamic collaboration of IoT devices.

[0025] The fifth aspect of this application provides a computer program product, wherein the computer program product includes a computer program that, when executed by a processor, implements the steps of the IoT device dynamic collaboration method as described above.

[0026] Beneficial effects: 1. Based on the current observation state of the target device and the actual actions of the cooperating device, this application dynamically selects a target strategy for the target device from the strategy knowledge base, and controls the target device based on the selected target strategy. This realizes the dynamic selection of control strategy for the target device, avoiding the problem of poor control effect caused by fixed strategies being unable to adapt to complex and ever-changing environments.

[0027] 2. After obtaining the target strategy, this application will obtain the strategy preference corresponding to the target strategy, and then use the strategy preference to adjust the initial action determined based on the current environmental state, thereby improving the accuracy of the predicted current action and enabling the target device to better match the expected goal when performing the action.

[0028] 3. This application updates confidence weights and selects target strategies based on the loss difference between predicted and actual actions. This allows for rapid identification of which strategies are more effective in new scenarios, enabling timely switching to the appropriate target strategy and ensuring stable operation and effective control of the target device. Even in complex and changing environments, it can react quickly and adjust control strategies, avoiding control failures or abnormal situations caused by unsuitable strategies.

[0029] 4. This application utilizes the automatic optimization of the number of clusters based on the silhouette coefficient, and then uses the number of clusters to cluster the potential vectors to form several candidate strategies. This achieves unsupervised construction of candidate strategies, avoids the subjectivity and limitations caused by manual intervention, and improves the efficiency and accuracy of strategy construction. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart illustrating an application scenario of the IoT device dynamic collaboration method provided in this application embodiment.

[0032] Figure 2A schematic block diagram of an electronic device provided in an embodiment of this application.

[0033] Figure 3 A flowchart of the dynamic collaboration method for IoT devices provided in this application embodiment.

[0034] Figure 4 This is a flowchart illustrating an example of the dynamic collaboration method for IoT devices provided in an embodiment of this application.

[0035] Figure 5 This is a flowchart illustrating the process of building a strategy knowledge base.

[0036] Figure 6 This is a flowchart illustrating the process of determining several potential variable clusters.

[0037] Figure 7 A flowchart illustrating the process of determining the predicted action for each candidate strategy.

[0038] Figure 8 This is a flowchart illustrating the process of determining the loss difference.

[0039] Figure 9 This is a flowchart illustrating an example of the process for selecting a target strategy.

[0040] Figure 10 This is a flowchart illustrating the process of determining confidence weights.

[0041] Figure 11 A flowchart illustrating the process of determining the current action of the target device.

[0042] Figure 12 This is a flowchart illustrating the process of determining strategy preferences.

[0043] Figure 13 A schematic diagram of the dynamic coordination device for IoT devices provided in the embodiments of this application. Detailed Implementation

[0044] This application provides a method, apparatus, device, storage medium, and product for dynamic collaboration of IoT devices. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description, with reference to the accompanying drawings and embodiments, further illustrates this application. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit this application.

[0045] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0046] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0047] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution sequence of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0048] Research has revealed that IoT technology and applications are still booming and have been widely applied in dynamic collaborative scenarios within the Industrial Internet. In these scenarios, collaborative IoT devices typically operate using static strategies. However, when unexpected situations arise in collaborative tasks, static strategies often fail to respond and adjust in a timely manner, leading to low collaborative efficiency and difficulty in meeting the complex and ever-changing demands of industrial production. For example, if a critical piece of equipment on the production line suddenly malfunctions, other collaborative IoT devices operating according to static strategies cannot quickly adjust their operating modes and will continue production as planned. This not only wastes resources but may also cause the entire production process to stall, affecting product delivery time and quality.

[0049] To address the aforementioned issues, this application embodiment acquires the current environmental state of the target device and the actual actions of the collaborating device; predicts the predicted actions of the collaborating device under each candidate strategy in the strategy knowledge base based on the current environmental state; selects a target strategy from the strategy knowledge base based on the predicted actions and the actual actions; determines the current action of the target device based on the target strategy and the current environmental state; and controls the target device to execute the current action. This application dynamically selects a target strategy for the target device from the strategy knowledge base based on the current observed state of the target device and the actual actions of the collaborating device, and controls the target device based on the selected target strategy. This achieves dynamic selection of control strategies for the target device, avoiding the problem of poor control performance caused by fixed strategies being unable to adapt to complex and changing environments.

[0050] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0051] Please see Figure 1 , Figure 1 This is an application environment diagram of the dynamic collaboration method for IoT devices provided in the embodiments of this application. For example... Figure 1 As shown in the diagram, the IoT device dynamic collaboration system in this application environment includes a cloud device 110 and a target device 120. The cloud device 110 and the target device 120 are connected via a network. The cloud device 110 can be implemented using a standalone server or a server cluster consisting of multiple servers. The target device 120 can specifically be a desktop terminal device or a mobile terminal device; the mobile terminal device can specifically be at least one of a mobile phone, tablet computer, or laptop computer. Specifically, the target device 120 obtains a policy knowledge base from the cloud device 110, acquires its current environmental state and the actual actions of the collaborating devices; based on the current environmental state, it predicts the predicted action of the collaborating device under each candidate policy in the policy knowledge base, and selects a target policy from the policy knowledge base based on the predicted action and the actual action; based on the target policy and the current environmental state, it determines the current action of the target device and controls the target device to execute the current action.

[0052] Please see Figure 2 , Figure 2 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Figure 2 As shown, the electronic device includes a processor and a memory. The processor 1001 and the memory 1002 are connected by a bus, such as an I2C (Inter-integrated Circuit) bus or a distributed soft bus.

[0053] The memory 1002 may include a non-volatile storage medium and internal memory. The non-volatile storage medium may store an operating system and a computer program. The computer program includes program instructions that, when executed, cause the processor to perform any dynamic coordination method for Internet of Things (IoT) devices.

[0054] Processor 1001 provides computing and control capabilities to support the operation of the entire electronic device. Processor 1001 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0055] Specifically, the processor 1001 is used to run a computer program stored in memory, and when executing the computer program, it performs the following steps: Obtain the current environmental status of the target device and the actual actions of the collaborating devices; Based on the current environmental state, predict the predicted action of the collaborative device under each candidate strategy in the strategy knowledge base, and select the target strategy in the strategy knowledge base based on the predicted action and the actual action. Based on the target strategy and the current environmental state, the current action of the target device is determined, and the target device is controlled to execute the current action.

[0056] In some embodiments, the candidate strategy includes several latent variables, which are constructed based on trajectory data, wherein each trajectory point in the trajectory data includes an environmental state and an action.

[0057] In some embodiments, when the processor 1001 predicts the predicted action of the collaborative device under each candidate policy in the policy knowledge base based on the current environmental state, it is specifically configured to implement: For each latent variable in the candidate strategy, a first conditional distribution probability of the predicted action corresponding to the latent variable is determined based on the current environmental state. The predicted action with the highest probability in the first conditional distribution is selected to obtain the predicted action of the collaborative device under the candidate strategy.

[0058] In some embodiments, when the processor 1001 selects a target policy from the policy knowledge base based on the predicted action and the actual action, it specifically implements the following: Obtain the loss difference between the predicted action and the actual action under each candidate strategy; The confidence weights of candidate strategies are determined based on all loss differences, and the target strategy is selected from the strategy knowledge base according to the confidence weights.

[0059] In some embodiments, when the processor 1001 obtains the loss difference between the predicted action and the actual action under each candidate policy, it is specifically configured to: Read the latent variables corresponding to the predicted action under each candidate strategy in the strategy knowledge base; Based on the current environmental state, determine the second conditional distribution probability of the actual action under each latent variable read; The loss difference between the predicted action and the actual action under each candidate policy is determined based on the second conditional distribution probability under each latent variable.

[0060] In some embodiments, when the processor 1001 determines the confidence weights of candidate strategies based on all loss differences, it is specifically configured to implement: The current confidence weight of each candidate policy in the policy knowledge base is updated exponentially based on all loss differences to obtain the updated confidence weight. The updated confidence weights are smoothed based on the smoothing parameters corresponding to the candidate strategies to obtain the confidence weights of the candidate strategies.

[0061] In some embodiments, each candidate strategy uses the same smoothing parameters in the smoothing process.

[0062] In some embodiments, when the processor 1001 determines the current action of the target device based on the target policy and the current environmental state, it is specifically configured to implement: Determine the initial action distribution of the target device based on the current environmental state; Obtain the policy preference corresponding to the target policy, and adjust the initial action distribution based on the policy preference to obtain the target action distribution; The current action of the target device is determined based on the target action distribution.

[0063] In some embodiments, when the processor 1001 acquires the policy preference corresponding to the target policy, it is specifically configured to: Obtain the policy category of the target policy; Based on the strategy category, select strategy preferences for the target strategy.

[0064] In some embodiments, when constructing the policy knowledge base, the processor 1001 is specifically used to implement: Acquire trajectory data, wherein each trajectory point in the trajectory data includes environmental state and action; The latent variable distribution parameters of the trajectory points in the trajectory data are obtained by a variational autoencoder. The latent variable set is determined based on the latent variable distribution parameters, and the latent variable set is clustered into several latent variable clusters; Each cluster of latent variables is treated as a candidate policy, resulting in a policy knowledge base.

[0065] In some embodiments, when determining the set of latent variables based on the latent variable distribution parameters, the processor 1001 is specifically configured to: The distribution parameters of the latent variables corresponding to each trajectory point are reparameterized to obtain the latent variables corresponding to each trajectory point; The set of latent variables corresponding to all trajectory points is called the latent variable set.

[0066] In some embodiments, when the processor 1001 clusters the latent variables into several latent variable clusters, it is specifically configured to: The number of clusters is determined based on the silhouette coefficient, which is determined based on the average intra-cluster distance and the nearest neighbor cluster distance. The latent variable set is clustered based on the number of clusters to obtain several latent variable clusters.

[0067] In some embodiments, after treating each latent variable cluster as a candidate strategy to obtain several candidate strategies, the processor 1001 is further configured to implement: Obtain the policy anchor point for each candidate policy; The policy anchor is obtained to determine the policy category of the candidate policy.

[0068] In some embodiments, the processor 1001 is further configured to implement: Construct new trajectory points based on the current environment state and the current action; The strategy knowledge base is updated based on the new trajectory points.

[0069] In some embodiments, when the processor 1001 updates the policy knowledge base based on the new trajectory points, it is specifically configured to implement: Obtain the new latent variables corresponding to the new trajectory points, and determine the candidate strategies to which the new latent variables belong; The policy anchor of the candidate policy is updated based on the new latent variables to update the policy knowledge base.

[0070] In some embodiments, the process of updating the policy anchor point of the candidate policy based on the new latent variable is as follows: , , in, This indicates the updated policy anchor. This indicates the policy anchor point before the update. Indicates a new latent variable. Indicates the learning rate. This indicates the number of latent variables contained in the latent variable cluster before the addition of the new latent variable.

[0071] In one embodiment, the processor 1001 is used to receive a policy knowledge base transmitted from a cloud device, wherein the construction and updating processes of the policy knowledge base are executed by the cloud device.

[0072] Please see Figure 3 , Figure 3 This is a flowchart of the dynamic coordination method for IoT devices provided in this embodiment. The method includes acquiring the current environmental state of the target device and the actual actions of the cooperating device; predicting the predicted action of the cooperating device under each candidate strategy in a strategy knowledge base based on the current environmental state; selecting a target strategy from the strategy knowledge base based on the predicted action and the actual action; determining the current action of the target device based on the target strategy and the current environmental state; and controlling the target device to execute the current action. This application dynamically selects a target strategy for the target device from the strategy knowledge base based on the current observed state of the target device and the actual actions of the cooperating device, and controls the target device based on the selected target strategy. This achieves dynamic selection of control strategies for the target device, avoiding the problem of poor control performance caused by fixed strategies being unable to adapt to complex and changing environments.

[0073] like Figure 3 and Figure 4 As shown, the IoT device dynamic collaboration method provided in this application embodiment specifically includes: S10. Obtain the current environmental status of the target device and the actual actions of the cooperating devices.

[0074] Specifically, the current environmental state refers to the observation data of the target device at the current moment. This current environmental state can include the target device's own physical state and external environmental information. The physical state can include basic operating parameters such as temperature, pressure, and rotational speed. External environmental information can include external factors that affect the operation of the target device, such as temperature, humidity, and light intensity. Furthermore, the current environmental state can also include interaction information with other collaborating devices. This interaction information can include the location, operating status, and communication status of the collaborating devices with the target device.

[0075] Collaborating devices are other devices that work in conjunction with the target device. The actual actions of a collaborating device refer to the operations that these other devices are performing at the current moment. These actions can be obtained through interaction between the target device and the collaborating device. For example, in a multi-robot emergency response system, the target device is a patrol robot responsible for patrolling, while the collaborating device is a firefighting robot responsible for extinguishing fires. During its patrol, the patrol robot acquires information about its current environmental conditions, such as the surrounding temperature and smoke concentration, and also obtains information about the firefighting robot's actual actions, such as whether it is spraying water to extinguish the fire.

[0076] S20. Based on the current environmental state, predict the predicted action of the collaborative device under each candidate strategy in the strategy knowledge base, and select the target strategy in the strategy knowledge base based on the predicted action and the actual action.

[0077] Specifically, the strategy knowledge base is pre-built and includes multiple candidate strategies, each of which is a cluster of latent variables. That is, each candidate strategy includes several latent variables, and each latent variable is applied to a trajectory point in the trajectory data used to construct the strategy knowledge base. This trajectory point includes environmental state and action, and the latent variables are constructed by using a variational autoencoder to perform variational encoding on the environmental state and action included in the trajectory point. This application, by using a variational autoencoder to construct latent variables, can effectively extract the feature information of environmental state and action in the trajectory point, mapping the high-dimensional original data to a low-dimensional latent space, thereby reducing data complexity.

[0078] It should be noted that the policy knowledge base can be built by the target device itself, or it can be built and deployed on the target device by an external device. For example, the policy knowledge base is built and maintained by a cloud device, and the target device obtains the policy knowledge base by communicating with the motion device and then deploys the policy knowledge base locally.

[0079] In one embodiment, such as Figure 5 As shown, the construction process of the strategy knowledge base specifically includes: H10. Obtain trajectory data. Each trajectory point in the trajectory data includes the environmental state and action. H20. Obtain the latent variable distribution parameters of the trajectory points in the trajectory data through a variational autoencoder; H30. Determine the set of latent variables based on the distribution parameters of the latent variables, and cluster the set of latent variables into several clusters of latent variables; H40. Treat each cluster of latent variables as a candidate policy to obtain a policy knowledge base.

[0080] In step H10, the trajectory data includes a running record over a period of time. The running record includes trajectory points at multiple points in time, and each trajectory point includes an environmental state and an action, where the action is the operation performed under the environmental state. The trajectory data can be represented as follows: , in, Represents trajectory data, express The environmental state at any given time, express Actions at any moment Indicates the number of moments.

[0081] It should be noted that after acquiring the trajectory data, a standardization operation is performed on the trajectory data according to a preset sliding window to eliminate the influence of different dimensions. The window length of the preset sliding window can be adjusted according to the actual situation; for example, it can be set to 50 time steps. The standardization operation can use Z-score normalization, which involves subtracting the mean of the trajectory points within the preset sliding window from each trajectory point, and then dividing by the standard deviation of the trajectory points within the preset sliding window.

[0082] Therefore, standardized operation can be expressed as: , , in, Represents the standardized trajectory points. This represents the mean of the preset sliding window. This represents the standard deviation of the preset sliding window. This indicates the trajectory points included in the preset sliding window. This indicates the window length of the preset sliding window.

[0083] In step H20, since the variational autoencoder is a deep learning module, it can learn the latent structure of the data. Therefore, the variational autoencoder encodes the trajectory points in the trajectory data to learn the latent structure of the trajectory points, thereby outputting the latent variable distribution parameters of the trajectory points. Specifically, the process of obtaining the latent variable distribution parameters of the trajectory points in the trajectory data through the variational autoencoder involves mapping the high-dimensional trajectory points to a low-dimensional latent space, thereby prompting the feature data of the trajectory points to obtain the latent variable distribution parameters. These latent variable distribution parameters reflect and describe the distribution of latent variables in the latent space, providing a basis for subsequently determining the latent variable set.

[0084] It should be noted that the variational autoencoder is trained. During the training process, the variational autoencoder uses the ELBO (Evidence Lower Bound) loss function in variational inference. This loss function compresses the device behavior pattern into a low-dimensional latent space while preserving the temporal features of the policy.

[0085] In step H30, the latent variable set includes the latent variables corresponding to each trajectory point, and each of the several latent variable clusters includes a subset of the latent variables from the latent variable set. Specifically, the latent variables corresponding to each trajectory point are obtained by sampling the distribution parameters of the latent variables corresponding to each trajectory point.

[0086] In one implementation, determining the latent variable set based on the latent variable distribution parameters specifically includes: The distribution parameters of the latent variables corresponding to each trajectory point are reparameterized to obtain the latent variables corresponding to each trajectory point; The set of latent variables corresponding to all trajectory points is called the latent variable set.

[0087] Specifically, reparameterization is used to sample the latent variable distribution parameters. That is, the latent variable distribution parameters corresponding to each trajectory point are reparameterized to obtain the latent variable corresponding to each trajectory point. Then, the set of latent variables corresponding to all trajectory points is taken as the latent variable set, such that the latent variable set includes the latent variable corresponding to each trajectory point.

[0088] Furthermore, after obtaining the set of latent variables, clustering (such as K-Means clustering) is performed on the latent variable set to group multiple latent variables into several latent variable clusters. The number of clusters for each latent variable cluster can be preset, such as manually setting the number of clusters to 5, or it can be determined based on the association relationships between latent variables, such as using the silhouette coefficient to determine the number of clusters.

[0089] In one implementation, such as Figure 6 As shown, the step of clustering the latent variables into several latent variable clusters specifically includes: H31. Determine the number of clusters based on the silhouette coefficient; H32. Cluster the set of latent variables based on the number of clusters to obtain several clusters of latent variables.

[0090] In step H31, the silhouette coefficient is used to measure the similarity between each latent variable and its cluster, as well as its separation from other latent variable clusters. The silhouette coefficient value ranges from -1 to 1. The closer the silhouette coefficient value is to 1, the higher the similarity between the latent variable and its cluster, and the greater the separation from other latent variable clusters. The silhouette coefficient is determined based on the intra-cluster average distance and the nearest neighbor cluster distance. The formula for calculating the silhouette coefficient can be expressed as: , in, Represents the cluster centers of latent variable clusters. The silhouette coefficients represent the clusters of latent variables. Indicates the average distance within the cluster. This represents the nearest neighbor cluster distance.

[0091] Furthermore, when determining the number of clusters based on the silhouette coefficient, a default number of clusters can be randomly generated. This default number of clusters is used as the initial number of clusters to cluster the latent variable set, resulting in initial latent variable clusters. Then, the silhouette coefficient of each initial latent variable cluster is calculated using the aforementioned formula, and it is checked whether the silhouette coefficient of each initial latent variable cluster meets the preset requirements. When there are initial latent variable clusters whose silhouette coefficients do not meet the preset requirements, the initial number of clusters is adjusted to obtain the adjusted initial number of clusters. This adjusted initial number of clusters is used as the initial number of clusters, and the variable set is re-clustered until the silhouette coefficients of all initial latent variable clusters meet the preset requirements, or the number of clustering operations reaches a preset threshold.

[0092] Specifically, when the termination condition is that the silhouette coefficients of all initial latent variable clusters meet the preset requirements, the initial number of clusters when the silhouette coefficients of all initial latent variable clusters meet the preset requirements is taken as the number of clusters; when the number of clustering reaches the preset number threshold, the number of clusters is selected based on the optimal silhouette coefficient among the initial number of clusters in all processes. The optimal silhouette coefficient can be the minimum number of initial latent variable clusters that do not meet the preset requirements, or the maximum average silhouette coefficient of all initial latent variable clusters, etc.

[0093] It should be noted that when adjusting the initial cluster number, it can be done manually based on experience, or by using a trained cluster number prediction model. The initial cluster number and the silhouette coefficient corresponding to each initial latent variable cluster are input into the cluster number prediction model, which then outputs the adjusted initial cluster number. The cluster number prediction model can be constructed using various machine learning algorithms, such as decision tree algorithms, neural network algorithms, and support vector machine algorithms.

[0094] In step H32, after obtaining the number of clusters, a clustering operation is performed on the latent variable set based on this number to obtain a number of latent variable clusters. Here, K-Means clustering is used as an example to illustrate the clustering process. Specifically, the K-Means clustering process is as follows: random initialization... Cluster centers ( The number of clusters is determined based on the silhouette coefficient. Then, each latent variable is assigned to the cluster containing the nearest cluster center. The cluster center of each cluster is then recalculated. This process of assigning and updating cluster centers is repeated until the cluster centers no longer change significantly or the preset number of iterations is reached, in order to obtain several latent variable clusters.

[0095] Furthermore, in step H40, after obtaining several clusters of latent variables, each cluster represents a possible device control strategy pattern, containing a set of latent variables with similar characteristics. These latent variables with similar characteristics reflect the device's behavioral characteristics under different environmental states. Therefore, each cluster of latent variables can be considered as a candidate strategy, and all candidate strategies constitute a strategy knowledge base. In other words, the strategy knowledge base includes several clustered device control strategy patterns.

[0096] This application's embodiments dynamically select the number of clusters based on the silhouette coefficient, and then cluster the latent variable set based on this number to form several candidate strategies, thereby creating a strategy knowledge base. This automatically adapts to the characteristics of different device behavior data, improving the accuracy and effectiveness of the strategy knowledge base construction. Compared to traditional methods with a fixed number of clusters, this method better captures latent patterns in the data, enabling the strategy knowledge base to more accurately reflect the device's action characteristics under various environmental conditions. It provides a rich selection of appropriate control strategies for the target device under different environmental conditions. For example, when the device is in a specific environmental state, the most suitable device control strategy mode can be quickly matched from the strategy knowledge base, thereby achieving efficient and stable operation of the device. Simultaneously, as device operation data accumulates, trajectory data can be periodically processed and the strategy knowledge base updated to adapt to dynamic changes in device behavior patterns.

[0097] In one embodiment, to better obtain the control type of each candidate policy, after obtaining several candidate policies, a policy category can be configured for each candidate policy, and this policy category reflects the control type of the candidate policy. Based on this, after treating each latent variable cluster as a candidate policy to obtain several candidate policies, the method further includes: Obtain the policy anchor point for each candidate policy; The policy anchor is obtained to determine the policy category of the candidate policy.

[0098] Specifically, the strategy anchor can be the cluster center in the latent variable cluster corresponding to the candidate strategy. That is, the cluster center in the latent variable cluster is used as the strategy anchor of the candidate strategy. Then, the control type corresponding to the cluster center is obtained and the control type is used as the strategy category of the candidate strategy. The control type can include aggressive, conservative, or stable, etc., and the strategy category can also include aggressive, conservative, or stable, etc.

[0099] It should be noted that when obtaining the control type corresponding to the cluster center, the latent variable corresponding to the cluster center is input into the decoder, the decoder outputs the action corresponding to the latent variable, and then the control type is determined based on the action and the action at the previous moment. For example, the movement distance between the action and the action at the previous moment is calculated, and the control type is determined based on the movement distance; or, the action, the action at the previous moment, and the task type are input into the preset type prediction module, and the control type is output through the preset type prediction model, etc.

[0100] The above describes the construction process of the strategy knowledge base. Step S20 will be described below.

[0101] In step S20, each candidate strategy in the strategy knowledge base corresponds to a prediction action. This prediction action is based on the current environmental state of the target device and uses the candidate strategy to predict the cooperative devices of the target device. The prediction action can be determined based on the cluster center of the latent variable cluster corresponding to the candidate strategy and the current environmental state, or it can be determined using all latent variables in the latent variable cluster corresponding to the candidate strategy and the current environmental state, or it can be determined by randomly selecting a latent variable from the latent variable cluster corresponding to the candidate strategy and then using that latent variable and the current environmental state, etc.

[0102] In one embodiment, such as Figure 7 As shown, the specific steps of predicting the predicted action of the collaborative device under each candidate policy in the policy knowledge base based on the current environmental state include: S211. For each latent variable in the candidate strategy, determine the first conditional distribution probability of the predicted action corresponding to the latent variable based on the current environmental state. S212. Select the predicted action with the highest probability of the first conditional distribution to obtain the predicted action of the collaborative device under the candidate strategy.

[0103] Specifically, the first conditional distribution probability is used to reflect the likelihood of the predicted action corresponding to the latent variable occurring under the current environmental state. The larger the first conditional distribution probability, the higher the likelihood of the predicted action corresponding to the latent variable occurring under the current environmental state; conversely, the smaller the first conditional distribution probability, the lower the likelihood of the predicted action corresponding to the latent variable occurring under the current environmental state.

[0104] The first conditional probability distribution is determined by a decoder whose inputs include latent variables and the current environment state. The latent variables serve as input data providing features, and the current environment state serves as a conditional constraint. Therefore, the first conditional probability distribution can be expressed as: , in, This represents the probability of the first conditional distribution. The model parameters are: decoder, Indicates a predicted action. Representing latent variables, Indicates the current environmental status, Furthermore, after obtaining the first conditional distribution probability of each latent variable in the candidate strategy, the predicted action of the collaborative device under the candidate strategy is selected from all obtained predicted actions based on the highest first conditional distribution probability. The predicted action of the collaborative device under the candidate strategy can be expressed as: , in, This indicates the predicted action of the collaborative device under the candidate strategy. This represents the cluster of latent variables corresponding to the candidate strategy.

[0105] This application's embodiments compare the first conditional distribution probabilities corresponding to each latent variable, and then select the predicted action with the highest first conditional distribution probability as the predicted action of the collaborative device under the candidate strategy. This allows for more accurate selection of the predicted action that best matches the current environment and the characteristics of the latent variables from multiple predicted actions. This helps the collaborative device make more reasonable and effective predicted actions based on the candidate strategy in complex and ever-changing environments, thereby improving the working efficiency and adaptability of the collaborative device and better completing collaborative tasks with other devices or systems.

[0106] Furthermore, in step S20, after obtaining the predicted actions of the collaborative device under each candidate strategy, a target strategy is selected from the strategy knowledge base based on the predicted actions and actual actions of the collaborative device under each candidate strategy. The selection of the target strategy from the strategy knowledge base can be achieved through various methods. For example, calculating the similarity between the predicted and actual actions (such as Euclidean distance, Manhattan distance, cosine similarity, etc.) and then selecting the target strategy based on the similarity; or calculating the loss difference between the predicted and actual actions (such as log-likelihood loss, cross-entropy loss, etc.) and then selecting the target strategy based on the loss difference; or calculating the loss difference between the predicted and actual actions, adjusting the confidence weight of the candidate strategy based on the loss difference, and then selecting the target strategy based on the adjusted confidence weight.

[0107] In one embodiment, a target strategy is selected based on the Euclidean distance between the predicted action and the actual action. Accordingly, the selection of the target strategy from the strategy knowledge base based on the predicted action and the actual action specifically includes: For each candidate policy in the policy knowledge base, calculate the Euclidean distance between the predicted action and the actual action of the cooperative device under that candidate policy; The candidate policy with the smallest Euclidean distance is selected from the policy knowledge base as the target policy.

[0108] This application's embodiments select the target strategy based on the Euclidean distance between the predicted action and the actual action of the collaborative device under the candidate strategy. This allows for a more intuitive and effective measurement of the difference between the predicted and actual actions. Furthermore, selecting the target strategy based on Euclidean distance is computationally simple and efficient, enabling the rapid selection of the most suitable strategy from numerous candidate strategies. This is beneficial in practical applications, especially in scenarios with high real-time requirements, for quickly determining appropriate control strategies for collaborative devices, ensuring their stable and efficient operation.

[0109] In one embodiment, a target strategy is selected based on the loss difference between the predicted action and the actual action. Accordingly, selecting the target strategy from the strategy knowledge base based on the predicted action and the actual action specifically includes: Obtain the loss difference between the predicted action and the actual action under each candidate strategy; The candidate strategy with the smallest loss difference is selected from the strategy knowledge base as the target strategy.

[0110] Specifically, loss difference measures the degree of deviation between the predicted action and the actual action. The smaller the loss difference, the closer the predicted action is to the actual action, the higher the accuracy of the candidate strategy, and correspondingly, the greater its confidence weight. Conversely, the larger the loss difference, the greater the deviation between the predicted action and the actual action, the lower the reliability of the candidate strategy, and the smaller its confidence weight. Loss difference can be calculated using various loss functions, such as log-likelihood loss and cross-entropy loss.

[0111] In one embodiment, such as Figure 8 As shown, obtaining the loss difference between the predicted action and the actual action under each candidate strategy specifically includes: M10. Read the latent variables corresponding to the predicted actions under each candidate strategy in the strategy knowledge base; M20. Based on the current environmental state, determine the second conditional distribution probability of the actual action under each latent variable read; M30. Determine the loss difference between the predicted action and the actual action under each candidate policy based on the second conditional distribution probability under each latent variable.

[0112] Specifically, the latent variables are the potential variables used by the candidate strategy to determine the preset action, and the second conditional distribution probability reflects the probability of the actual action occurring under the current environmental state and the latent variables. The decoder's input includes the latent variables and the current environmental state, and its output is the probability of the actual action occurring under the given latent variables. Based on the second conditional distribution probability, an appropriate loss function is used to calculate the loss difference between the predicted action and the actual action.

[0113] For example, when the loss function is the log-likelihood loss function, the second conditional distribution probability can be substituted into the calculation formula of the log-likelihood loss to obtain the corresponding loss value, and the loss difference between the predicted action and the actual action can be expressed. This loss difference intuitively reflects the deviation between the predicted action and the actual action; the smaller the loss difference, the closer the predicted action is to the actual action, and the higher the accuracy of the candidate strategy.

[0114] In this embodiment, the target strategy is selected based on minimizing the difference in loss. This allows for a more accurate assessment of the reliability and accuracy of candidate strategies, enabling the selection of the strategy that best reflects the actual situation from among numerous candidates. This helps collaborative devices operate based on more reliable strategies when performing tasks, reducing errors or inefficiencies caused by inaccurate strategies.

[0115] In one embodiment, the target policy is selected based on the confidence weights of the candidate policies. Accordingly, such as... Figure 9As shown, the selection of the target strategy from the strategy knowledge base based on the predicted action and the actual action specifically includes: S221. Obtain the loss difference between the predicted action and the actual action under each candidate strategy; S222. Determine the confidence weights of candidate strategies based on all loss differences, and select the target strategy from the strategy knowledge base according to the confidence weights.

[0116] Specifically, the process of obtaining the loss difference between the predicted action and the actual action under each candidate strategy is the same as the process of obtaining the loss difference between the predicted action and the actual action under each candidate strategy in the above embodiments, and will not be repeated here.

[0117] The confidence weight is determined based on all loss differences identified at the current time. It can be calculated using all loss differences or determined by updating the confidence weights configured for the candidate strategy itself using all loss differences. The confidence weight is used as the basis for selecting the target strategy. For example, after obtaining the confidence weight of each candidate strategy, the candidate strategy with the highest confidence weight can be selected as the target strategy.

[0118] In one embodiment, such as Figure 10 As shown, the confidence weights for determining candidate strategies based on all loss differences specifically include: S2221. Based on all loss differences, the current confidence weight of each candidate policy in the policy knowledge base is updated exponentially to obtain the updated confidence weight. S2222: The updated confidence weights are smoothed based on the smoothing parameters corresponding to the candidate strategies to obtain the confidence weights of the candidate strategies.

[0119] Specifically, each candidate strategy in the strategy knowledge base is configured with a confidence weight, which is the confidence weight determined in the previous time step. In other words, the current confidence weight of each candidate strategy in the strategy knowledge base is the confidence weight determined in the previous time step.

[0120] Furthermore, the update process of the index can be represented as: , in, This represents the confidence weight after the index is updated. Indicates the current confidence weight. Indicates the first The difference in loss among candidate strategies No. The difference in loss among candidate strategies This represents the learning rate parameter.

[0121] Furthermore, after obtaining the updated confidence weights, the updated confidence weights can be smoothed using the smoothing parameters corresponding to the candidate strategies. The smoothing process can be expressed as follows: , in, This represents the confidence weight of the candidate strategy. Indicates the smoothing parameter. This indicates the number of candidate strategies in the strategy knowledge base.

[0122] It should be noted that in this application, the smoothing parameter and learning rate parameter are the same for each candidate policy in the policy knowledge base. For example, =0.2, =0.4.

[0123] This application calculates the loss difference for each candidate strategy, then updates the current confidence weight of each candidate strategy based on all loss differences to obtain the confidence weight of the candidate strategy. Finally, a target strategy is selected from the strategy knowledge base based on the confidence weight (e.g., the maximum confidence weight). This allows for a more dynamic and comprehensive evaluation of the merits of candidate strategies. Since the update of the confidence weight comprehensively considers all loss differences, it reflects the performance of each candidate strategy at different times, making the selected target strategy more closely aligned with changes in the actual scenario. This helps collaborative devices continuously adjust and optimize their adopted strategies during long-term operation, improving their adaptability and coping capabilities to complex environments. Simultaneously, smoothing processes avoid drastic fluctuations in confidence weights, making strategy adjustments more stable and reliable, reducing frequent strategy switching due to accidental factors, and ensuring the stability and efficiency of collaborative devices. Moreover, this confidence weight-based strategy selection method has good versatility and flexibility in different application scenarios, allowing adjustment of learning rate and smoothing parameters according to actual needs to achieve the best strategy selection effect, thereby better enabling collaborative devices to work collaboratively with other devices or systems.

[0124] S30 determines the current action of the target device based on the target strategy and the current environmental state, and controls the target device to execute the current action.

[0125] Specifically, after obtaining the target policy, the target policy is used as the control policy adopted by the target device in the current environmental state. Then, the current action of the target device is determined based on the target policy and the current golden state. The determination of the current action of the target device can be achieved by directly determining the action distribution based on the target policy; alternatively, the previously obtained policy categories can be used to adjust the action distribution determined based on the target policy, and then the current action of the target device can be determined based on the adjusted actions.

[0126] In one embodiment, such as Figure 11 As shown, determining the current action of the target device based on the target strategy and the current environmental state specifically includes: S31. Determine the initial action distribution of the target device based on the current environmental state; S32. Obtain the strategy preference corresponding to the target strategy, and adjust the initial action distribution based on the strategy preference to obtain the target action distribution; S33. Determine the current action of the target device based on the target action distribution.

[0127] In step S31, the initial action distribution is determined using the target strategy. The initial action distribution represents the probability distribution of actions that the target device may take under the current environmental conditions.

[0128] In step S32, strategy preferences are used to reflect the action characteristics or patterns favored by the target strategy. By applying strategy preferences to adjust the initial action distribution, the target action distribution can be made to better match the requirements of the target strategy, thereby more accurately determining the current action of the target device.

[0129] In one implementation, such as Figure 12 As shown, obtaining the strategy preference corresponding to the target strategy specifically includes: S321. Obtain the strategy category of the target strategy; S322. Select a strategy preference for the target strategy based on the strategy category.

[0130] Specifically, strategy categories are configured for each candidate strategy in the strategy knowledge base when it is built. Since the target strategy is a candidate strategy in the strategy knowledge base, it carries a strategy category. After obtaining the strategy category, a strategy preference can be selected for the target strategy based on that strategy category.

[0131] Specifically, when selecting a strategy preference for the target strategy based on the strategy category, a pre-set correspondence between strategy categories and strategy preferences can be used. Specifically, the strategy categories include conservative, stable, and aggressive categories, and the correspondence includes the correspondence between conservative category and strategy preference, the correspondence between stable category and strategy preference, and the correspondence between aggressive category and strategy preference.

[0132] For example: The correspondence between conservative categories and strategy preferences is as follows: Increase the probability of conservative actions by a first preset value, keep the probability of steady actions unchanged, and decrease the probability of aggressive actions by a second preset value.

[0133] The correspondence between the stable type and the strategy preference is as follows: Keep the probability of conservative actions unchanged, increase the probability of steady actions by a third preset value, and keep the probability of aggressive actions unchanged.

[0134] The correspondence between aggressive categories and strategy preferences is as follows: Decrease the probability of conservative actions by the fourth preset value, keep the probability of steady actions unchanged, and increase the probability of aggressive actions by the fifth preset value.

[0135] It should be noted that in practical applications, the strategy preference in step S32 can also be set according to the actual task, and there are no restrictions here.

[0136] Furthermore, after obtaining the policy preference, the initial action distribution is adjusted using the policy preference to obtain the target action distribution, which can be represented as: , in, Indicates the distribution of target actions. Indicate the target strategy, This represents the policy model corresponding to the target policy. Indicates policy preference, This represents the initial action distribution.

[0137] This application determines the initial action distribution based on the target strategy and the current environmental state, and then adjusts the initial action distribution using strategy preferences determined by the strategy category corresponding to the target strategy. This enables the target device to make action decisions more accurate. Because the target device's action decisions fully consider the specific circumstances of the current environment and the characteristics of the target strategy, it avoids blind decision-making and improves the accuracy and effectiveness of the target device in executing tasks. Simultaneously, this approach allows the target device to flexibly adjust its actions according to different target strategies and environmental states, enhancing its adaptability to unexpected situations in complex and changing environments and enabling it to better collaborate with other devices or systems to complete various tasks.

[0138] In one embodiment, after obtaining the current action of the target device, the policy knowledge base can be updated using the current environmental state and the current action, so that the policy knowledge base can better adapt to dynamic changes in the environment. Therefore, the IoT device dynamic collaboration method further includes: Construct new trajectory points based on the current environment state and the current action; The strategy knowledge base is updated based on the new trajectory points.

[0139] Specifically, when updating the policy knowledge base based on the new trajectory points, the new trajectory points can be added to the trajectory data to form new trajectory data, and then the policy knowledge base construction process can be re-executed to update the policy knowledge base. However, in practical applications, rebuilding the policy knowledge base for each execution not only requires a large amount of computing resources but also reduces the real-time performance of collaborative control.

[0140] Therefore, in one embodiment, when updating the policy knowledge base based on the new trajectory point, a new latent variable corresponding to the new trajectory point can be determined, and then the policy anchor point of the candidate policy in the policy knowledge base can be updated using the new latent variable to update the policy type of the candidate policy. Based on this, updating the policy knowledge base based on the new trajectory point specifically includes: Obtain the new latent variables corresponding to the new trajectory points, and determine the candidate strategies to which the new latent variables belong; The policy anchor of the candidate policy is updated based on the new latent variables to update the policy knowledge base.

[0141] Specifically, the process of obtaining new latent variables is the same as that of obtaining latent variables described above, and will not be repeated here. After obtaining the new latent variables, the distance between the new latent variables and the cluster centers of the latent policy clusters corresponding to the candidate policies can be calculated to determine the latent variable cluster to which the new latent variable belongs, and thus determine the candidate policy to which the new latent variable belongs.

[0142] After determining the candidate policy to which the new latent variable belongs, the policy anchor of the candidate policy is updated based on the new latent variable to update the policy knowledge base. The process of updating the policy anchor of the candidate policy based on the new latent variable is as follows: , , in, This indicates the updated policy anchor. This indicates the policy anchor point before the update. Indicates a new latent variable. Indicates the learning rate. This indicates the number of latent variables contained in the latent variable cluster before the addition of the new latent variable.

[0143] In summary, this embodiment provides the aforementioned dynamic coordination method for IoT devices. The method includes acquiring the current environmental state of the target device and the actual actions of the cooperating device; predicting the predicted action of the cooperating device under each candidate strategy in the strategy knowledge base based on the current environmental state; selecting a target strategy from the strategy knowledge base based on the predicted action and the actual action; determining the current action of the target device based on the target strategy and the current environmental state; and controlling the target device to execute the current action. This application dynamically selects a target strategy for the target device from the strategy knowledge base based on the current observed state of the target device and the actual actions of the cooperating device, and controls the target device based on the selected target strategy. This achieves dynamic selection of control strategies for the target device, avoiding the problem of poor control performance caused by fixed strategies being unable to adapt to complex and changing environments.

[0144] Based on the above-described method for dynamic collaboration of IoT devices, this embodiment provides a device for dynamic collaboration of IoT devices, such as... Figure 13 As shown, the IoT device dynamic collaboration device specifically includes: The acquisition module 100 is used to acquire the current environmental status of the target device and the actual actions of the cooperating devices. The selection module 200 is used to predict the predicted action of the collaborative device under each candidate strategy in the strategy knowledge base based on the current environmental state, and to select the target strategy in the strategy knowledge base based on the predicted action and the actual action. The control module 300 is used to determine the current action of the target device based on the target strategy and the current environmental state, and to control the target device to execute the current action.

[0145] Based on the above-described dynamic coordination method for IoT devices, this embodiment provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the dynamic coordination method for IoT devices in the above embodiment.

[0146] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, and portable compact disk read-only memory (CD-ROM). ROM: CD Read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0147] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0148] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an electronic device, acquire the current environmental state of the target device and the actual actions of the cooperating device; predict the predicted actions of the cooperating device under each candidate strategy in the strategy knowledge base based on the current environmental state, and select a target strategy in the strategy knowledge base based on the predicted actions and the actual actions; determine the current action of the target device based on the target strategy and the current environmental state, and control the target device to execute the current action.

[0149] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or target device. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0150] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0151] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0152] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described dynamic coordination method for IoT devices, thereby solving the technical problem of poor application performance. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the dynamic coordination method for IoT devices provided in the above embodiments, and will not be repeated here.

[0153] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described dynamic coordination method for Internet of Things devices.

[0154] The computer program product provided in this application can solve the technical problem of poor application performance. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the IoT device dynamic coordination method provided in the above embodiments, and will not be repeated here.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for dynamic collaboration of Internet of Things (IoT) devices, characterized in that, The aforementioned dynamic collaboration method for IoT devices specifically includes: Obtain the current environmental status of the target device and the actual actions of the collaborating devices; Based on the current environmental state, predict the predicted action of the collaborative device under each candidate strategy in the strategy knowledge base, and select the target strategy in the strategy knowledge base based on the predicted action and the actual action. Based on the target strategy and the current environmental state, the current action of the target device is determined, and the target device is controlled to execute the current action.

2. The method for dynamic collaboration of IoT devices according to claim 1, characterized in that, The candidate strategy includes several latent variables, which are constructed based on trajectory data. Each trajectory point in the trajectory data includes environmental state and action.

3. The dynamic collaboration method for IoT devices according to claim 1 or 2, characterized in that, The specific steps of predicting the predicted action of the collaborative device under each candidate policy in the policy knowledge base based on the current environmental state include: For each latent variable in the candidate strategy, a first conditional distribution probability of the predicted action corresponding to the latent variable is determined based on the current environmental state. The predicted action with the highest probability in the first conditional distribution is selected to obtain the predicted action of the collaborative device under the candidate strategy.

4. The method for dynamic collaboration of IoT devices according to claim 1, characterized in that, The selection of a target strategy from the strategy knowledge base based on the predicted action and the actual action specifically includes: Obtain the loss difference between the predicted action and the actual action under each candidate strategy; The confidence weights of candidate strategies are determined based on all loss differences, and the target strategy is selected from the strategy knowledge base according to the confidence weights.

5. The method for dynamic collaboration of IoT devices according to claim 4, characterized in that, The specific steps of obtaining the loss difference between the predicted action and the actual action under each candidate strategy include: Read the latent variables corresponding to the predicted action under each candidate strategy in the strategy knowledge base; Based on the current environmental state, determine the second conditional distribution probability of the actual action under each latent variable read; The loss difference between the predicted action and the actual action under each candidate policy is determined based on the second conditional distribution probability under each latent variable.

6. The method for dynamic collaboration of IoT devices according to claim 4, characterized in that, The confidence weights for determining candidate strategies based on all loss differences specifically include: The current confidence weight of each candidate policy in the policy knowledge base is updated exponentially based on all loss differences to obtain the updated confidence weight. The updated confidence weights are smoothed based on the smoothing parameters corresponding to the candidate strategies to obtain the confidence weights of the candidate strategies.

7. The method for dynamic collaboration of IoT devices according to claim 6, characterized in that, Each candidate strategy uses the same smoothing parameters in the smoothing process.

8. The method for dynamic collaboration of IoT devices according to claim 1, characterized in that, Determining the current action of the target device based on the target strategy and the current environmental state specifically includes: Determine the initial action distribution of the target device based on the current environmental state; Obtain the policy preference corresponding to the target policy, and adjust the initial action distribution based on the policy preference to obtain the target action distribution; The current action of the target device is determined based on the target action distribution.

9. The method for dynamic collaboration of IoT devices according to claim 8, characterized in that, The specific steps of obtaining the strategy preference corresponding to the target strategy include: Obtain the policy category of the target policy; Based on the strategy category, select strategy preferences for the target strategy.

10. The method for dynamic collaboration of IoT devices according to claim 1, characterized in that, The construction process of the strategy knowledge base specifically includes: Acquire trajectory data, wherein each trajectory point in the trajectory data includes environmental state and action; The latent variable distribution parameters of the trajectory points in the trajectory data are obtained by a variational autoencoder. The latent variable set is determined based on the latent variable distribution parameters, and the latent variable set is clustered into several latent variable clusters; Each cluster of latent variables is treated as a candidate policy, resulting in a policy knowledge base.

11. The method for dynamic collaboration of IoT devices according to claim 10, characterized in that, The determination of the latent variable set based on the latent variable distribution parameters specifically includes: The distribution parameters of the latent variables corresponding to each trajectory point are reparameterized to obtain the latent variables corresponding to each trajectory point; The set of latent variables corresponding to all trajectory points is called the latent variable set.

12. The method for dynamic collaboration of IoT devices according to claim 10, characterized in that, The step of clustering the latent variables into several latent variable clusters specifically includes: The number of clusters is determined based on the silhouette coefficient, which is determined based on the average intra-cluster distance and the nearest neighbor cluster distance. The latent variable set is clustered based on the number of clusters to obtain several latent variable clusters.

13. The method for dynamic collaboration of IoT devices according to claim 10, characterized in that, After treating each cluster of latent variables as a candidate strategy to obtain several candidate strategies, the method further includes: Obtain the policy anchor point for each candidate policy; The policy anchor is obtained to determine the policy category of the candidate policy.

14. The method for dynamic collaboration of IoT devices according to claim 1, characterized in that, The method further includes: Construct new trajectory points based on the current environment state and the current action; The strategy knowledge base is updated based on the new trajectory points.

15. The method for dynamic collaboration of IoT devices according to claim 14, characterized in that, The process of updating the strategy knowledge base based on the new trajectory points specifically includes: Obtain the new latent variables corresponding to the new trajectory points, and determine the candidate strategies to which the new latent variables belong; The policy anchor of the candidate policy is updated based on the new latent variables to update the policy knowledge base.

16. The method for dynamic collaboration of IoT devices according to claim 15, characterized in that, The process of updating the policy anchor point of the candidate policy based on the new latent variables is as follows: ， ， in, This indicates the updated policy anchor. This indicates the policy anchor point before the update. Indicates a new latent variable. Indicates the learning rate. This indicates the number of latent variables contained in the latent variable cluster before the addition of the new latent variable.

17. The method for dynamic collaboration of IoT devices according to claim 1, characterized in that, The construction and updating of the strategy knowledge base are performed by cloud devices.

18. A dynamic collaboration device for Internet of Things (IoT) devices, characterized in that, The aforementioned IoT device dynamic collaboration device specifically includes: The acquisition module is used to acquire the current environmental status of the target device and the actual actions of the collaborating devices. The selection module is used to predict the predicted action of the collaborative device under each candidate strategy in the strategy knowledge base based on the current environmental state, and to select the target strategy in the strategy knowledge base based on the predicted action and the actual action. The control module is used to determine the current action of the target device based on the target strategy and the current environmental state, and to control the target device to execute the current action.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the IoT device dynamic collaboration method as described in any one of claims 1 to 17.

20. An electronic device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the IoT device dynamic collaboration method as described in any one of claims 1 to 17.

21. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the IoT device dynamic collaboration method as described in any one of claims 1 to 17.

Images