Artificial intelligence-based digital twin agent spatiotemporal collaboration method and metaverse

Abstract

The digital twin agent spatiotemporal collaboration method and metaverse based on artificial intelligence include: preset scene acquisition module; twin agent construction module; twin agent spatiotemporal collaboration module; multi-twin agent reinforcement learning model construction module based on spatiotemporal cognitive differences; multi-twin agent reinforcement learning model construction module based on spatiotemporal consistency representation; multi-twin agent reinforcement learning model construction module based on spatiotemporal dynamic collaboration graph. The above method and metaverse combine digital twin technology with artificial intelligence technology, and propose twin agents, which have all the characteristics of digital twins and all the characteristics of agents, so that multi-agent technology can be seamlessly integrated into digital twin systems and metaverses such as unmanned factories; combine the spatiotemporal information of multi-twin agents for more effective communication, and improve the global cognition and collaborative decision-making capabilities of multi-twin agents.

Description

Technical Field

[0001] The present invention relates to the field of digital twin technology, and in particular to a digital twin intelligent body spatiotemporal collaboration method and a metaverse based on artificial intelligence. Background Art

[0002] In the process of realizing the present invention, the inventors found that there are at least the following problems in the prior art: most of the existing multi-intelligence collaborative decision-making technologies based on deep reinforcement learning directly perform deep reinforcement learning on the observation values ​​and ignore the influence of observation time and space and observation methods on the observation values ​​and their influence on collaborative decision-making. In fact, the influence of observation time and space and observation methods on the observation values ​​is objective and unavoidable. If the influence of such time and space differences is ignored and collaborative decision-making is only based on the observation values, the efficiency and effect of training and testing of the collaborative decision-making model will inevitably be reduced. Only by making full use of the knowledge of observation time and space and observation methods can more comprehensive information be provided for collaborative decision-making. At the same time, the spatiotemporal information of multiple agents in unmanned factories is closely related to the collaboration and interaction between multiple agents. If the spatiotemporal information of multiple agents is ignored and missing during deep reinforcement learning, it will cause the model to fit incorrectly and even lead to wrong results in collaborative decision-making. For example, if there are three intelligent agents A, B and C in an unmanned factory, agent A needs to collaborate with agent B or agent C. If there is spatial information and agent B is closer to agent A, then agent B will be correctly selected. Otherwise, the possibility and basis for selection will be lost, which will affect the result of collaborative decision-making.

[0003] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0004] Based on this, it is necessary to provide a digital twin intelligent agent spatiotemporal collaboration method and metaverse based on artificial intelligence to address the defects or shortcomings of existing technologies, so as to solve the problem that most of the existing multi-intelligence collaborative decision-making technologies based on deep reinforcement learning directly perform deep reinforcement learning on the observation values ​​while ignoring the impact of observation time and space and observation methods on the observation values ​​and their impact on collaborative decision-making.

[0005] In a first aspect, an embodiment of the present invention provides a twin agent collaboration method, the method comprising:

[0006] Preset scene acquisition steps: acquire the preset scene;

[0007] Steps for building a twin intelligent agent: build a twin intelligent agent based on the objects in the preset scene; the twin intelligent agent has the characteristics of an intelligent agent and also has the characteristics of a digital twin;

[0008] Twin agent spatiotemporal collaboration steps: collect data of each object in a preset scene through the Internet of Things, use it as the data of the twin agent of each object, and input the data into a preset deep reinforcement learning model; use the preset deep reinforcement learning model to perform spatiotemporal communication and collaborative decision-making between the twin agents of each object in the preset scene; the preset deep reinforcement learning model includes a multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences or / and a multi-twin agent reinforcement learning model represented by spatiotemporal consistency or / and a graph convolution multi-twin agent reinforcement learning model based on a spatiotemporal dynamic collaboration graph.

[0009] Preferably, the method further comprises:

[0010] The steps of constructing a multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences are as follows: the local observation values ​​of all twin agents in the preset scene are converted into an observation value cognitive matrix e through a neural network; e is an n×d-dimensional encoding matrix, where n is the number of twin agents, d is the size of the encoding dimension, and e represents the result of encoding the observation values ​​of all twin agents; the observation methods of all twin agents in the preset scene are converted into a reference cognitive matrix e' through a neural network; the observation methods of twin agents include the time of local observation, the spatial position of local observation, and the method of local observation; e' is an n d'× dimensional encoding matrix, where n is the number of twin agents, d' is the size of the observation method dimension, and e' represents the result of encoding the observation methods of all twin agents; the observation value cognitive query matrix Q, the observation value cognitive value matrix K, and the observation value cognitive value matrix V are set to be the same as the observation value cognitive matrix e; the difference between the unit matrix and the dot product of the matrices Q' and K' is calculated, and added to the dot product of the matrices Q and K; the result is normalized using a soft max operation to obtain the attention observation cognitive coefficient matrix Where dk is used to scale the calculated results; the attention observation cognitive coefficient matrix a and the observation value cognitive value matrix V are used to calculate the attention observation cognitive matrix Use the observation value cognitive matrix to subtract the attention observation cognitive matrix, and then fuse it with the observation value cognitive matrix e converted by the neural network to generate a new input enew =

[0011]

[0012] Preferably, the method further comprises:

[0013] The steps of building a multi-twin agent reinforcement learning model based on spatiotemporal consistency representation are as follows: a spatiotemporal cognitive coding network is created, and multiple twin agents share the parameters of the spatiotemporal cognitive coding network; the spatiotemporal cognitive coding network is trained by taking the hidden state hi of the current twin agent as input and the hidden states hj of other twin agents as targets; the loss function of the spatiotemporal cognitive coding network is Among them, ψ is the parameter of the spatiotemporal cognitive coding network, N is the number of twin agents, hj i rep is the output of the spatiotemporal cognitive coding network, which means that the cognition of twin agent j is converted into the spatiotemporal consistency representation of twin agent i; j represents a twin agent different from twin agent i; the similarity of the spatiotemporal consistency representation of the hidden state of each twin agent and the hidden state of other twin agents is used to represent its degree of cognition of the preset scene. Where i ranges from 1 to N, and j includes 0; the degree of cognition of all twin agents to the environment can be represented by the matrix W; Use W to calculate the individual reward of each twin agent; the larger the W value, the higher the reward given to the twin agent.

[0014] Preferably, the method further comprises:

[0015] The steps for constructing a graph convolution multi-twin agent reinforcement learning model based on spatiotemporal dynamic collaborative graph are as follows: the spatial information, temporal information, and content information of the twin agents are used to construct the temporal adjacency matrix M1, the spatial adjacency matrix M2, and the content adjacency matrix M3. The three adjacency matrices are intersected to obtain a comprehensive adjacency matrix M; X is used to represent the observation value encoding matrix of the twin agents, which is an n×d-dimensional matrix, where n is the number of twin agents, d is the size of the twin agent encoding dimension, and Q and K are the query matrix and key value matrix that are the same as X; the similarity between the twin agent observation value encodings is calculated through the self-attention mechanism Where dk is a normalization factor that scales the result. X' is used to represent the observation encoding matrix of the twin agents, which is an n×d'-dimensional matrix, where n is the number of twin agents, d' is the size of the twin agent observation encoding dimension, and Q' and K' are the same query matrix and key value matrix as X'. The similarity between the twin agent observation encodings is calculated through the self-attention mechanism. Where d'k is a normalization factor to scale the result; the obtained adjacency matrix M is used to filter out the similarity values ​​between twin agents that do not need to interact As the spatiotemporal interaction relationship between twin agents; the spatiotemporal interaction relationship between twin agents is used as an adjacency matrix for graph convolution to achieve information fusion between twin agents Where X (l)is the feature matrix of convolutional layer l, W (l) is a d×d dimensional trainable matrix of the lth convolutional layer, and σ is a nonlinear activation function.

[0016] In a second aspect, an embodiment of the present invention provides a twin agent collaboration system, the system comprising:

[0017] Preset scene acquisition module: obtain preset scenes;

[0018] Twin intelligent agent construction module: builds a twin intelligent agent based on the objects in the preset scene; the twin intelligent agent has the characteristics of an intelligent agent and also has the characteristics of a digital twin;

[0019] Twin agent spatiotemporal collaboration module: collect data of each object in a preset scene through the Internet of Things, use it as the data of the twin agent of each object, and input the data into a preset deep reinforcement learning model; use the preset deep reinforcement learning model to perform spatiotemporal communication and collaborative decision-making between the twin agents of each object in the preset scene; the preset deep reinforcement learning model includes a multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences or / and a multi-twin agent reinforcement learning model represented by spatiotemporal consistency or / and a graph convolution multi-twin agent reinforcement learning model based on a spatiotemporal dynamic collaboration graph.

[0020] Preferably, the system further comprises:

[0021] The construction module of the multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences: the local observation values ​​of all twin agents in the preset scene are converted into an observation value cognitive matrix e through a neural network; e is an n×d-dimensional encoding matrix, where n is the number of twin agents, d is the size of the encoding dimension, and e represents the result of the observation value encoding of all twin agents; the observation mode of all twin agents in the preset scene is converted into a reference cognitive matrix e' through a neural network; the observation mode of the twin agent includes the time of local observation, the spatial position of local observation, and the system of local observation; e' is an n d'× dimensional encoding matrix, where n is the number of twin agents, d' is the size of the observation mode dimension, and e' represents the result of the observation mode encoding of all twin agents; the observation value cognitive query matrix Q, the observation value cognitive value matrix K and the observation value cognitive value matrix V are set to be the same as the observation value cognitive matrix e; the difference between the unit matrix and the dot product of the matrices Q' and K' is calculated, and added to the dot product of the matrices Q and K; the result is normalized using the soft max operation to obtain the attention observation cognitive coefficient matrix Where dk is used to scale the calculated results; the attention observation cognitive coefficient matrix a and the observation value cognitive value matrix V are used to calculate the attention observation cognitive matrix Use the observation value cognitive matrix to subtract the attention observation cognitive matrix, and then fuse it with the observation value cognitive matrix e converted by the neural network to generate a new input enew =

[0022]

[0023] Preferably, the system further comprises:

[0024] The building module of multi-twin agent reinforcement learning model based on spatiotemporal consistency representation: a spatiotemporal cognitive coding network is created, and multiple twin agents share the parameters of the spatiotemporal cognitive coding network; the spatiotemporal cognitive coding network is trained by taking the hidden state hi of the current twin agent as input and the hidden states hj of other twin agents as targets; the loss function of the spatiotemporal cognitive coding network is Among them, ψ is the parameter of the spatiotemporal cognitive coding network, N is the number of twin agents, hj i rep is the output of the spatiotemporal cognitive coding network, which means that the cognition of twin agent j is converted into the spatiotemporal consistency representation of twin agent i; j represents a twin agent different from twin agent i; the similarity of the spatiotemporal consistency representation of the hidden state of each twin agent and the hidden state of other twin agents is used to represent its degree of cognition of the preset scene. Where i ranges from 1 to N, and j includes 0; the degree of cognition of all twin agents to the environment can be represented by the matrix W; Use W to calculate the individual reward of each twin agent; the larger the W value, the higher the reward given to the twin agent.

[0025] Preferably, the system further comprises:

[0026] The construction module of the graph convolution multi-twin agent reinforcement learning model based on spatiotemporal dynamic collaborative graph: the spatial information, temporal information and content information of the twin agents are used to construct the temporal adjacency matrix M1, the spatial adjacency matrix M2 and the content adjacency matrix M3. The three adjacency matrices are intersected to obtain the comprehensive adjacency matrix M; X is used to represent the observation value encoding matrix of the twin agents, which is an n×d-dimensional matrix, where n is the number of twin agents, d is the size of the twin agent encoding dimension, and Q and K are the query matrix and key value matrix that are the same as X; the similarity between the twin agent observation value encodings is calculated through the self-attention mechanism Where dk is a normalization factor that scales the result. X' is used to represent the observation encoding matrix of the twin agents, which is an n×d'-dimensional matrix, where n is the number of twin agents, d' is the size of the twin agent observation encoding dimension, and Q' and K' are the same query matrix and key value matrix as X'. The similarity between the twin agent observation encodings is calculated through the self-attention mechanism. Where d'k is a normalization factor to scale the result; the obtained adjacency matrix M is used to filter out the similarity values ​​between twin agents that do not need to interact As the spatiotemporal interaction relationship between twin agents; the spatiotemporal interaction relationship between twin agents is used as an adjacency matrix for graph convolution to achieve information fusion between twin agents Where X (l) is the feature matrix of convolutional layer l, W (l) is a d×d dimensional trainable matrix of the lth convolutional layer, and σ is a nonlinear activation function.

[0027] In the third aspect, an embodiment of the present invention provides a multi-twin intelligent agent collaborative digital twin device, which includes a module of any one of the modules described in the second aspect embodiment.

[0028] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect.

[0029] In the fifth aspect, an embodiment of the present invention provides a multi-twin intelligent collaborative robot system, comprising a memory, a processor, and an artificial intelligence robot program stored in the memory and executable on the processor, characterized in that when the processor executes the program, the steps of the method described in any one of the embodiments of the first aspect are implemented.

[0030] In the sixth aspect, an embodiment of the present invention provides a multi-twin intelligent collaborative metaverse, including a metaverse, characterized in that the metaverse implements the steps of the method described in any one of the embodiments of the first aspect.

[0031] The digital twin agent spatiotemporal collaboration method and metaverse based on artificial intelligence provided in this embodiment include: a preset scene acquisition module; a twin agent construction module; a twin agent spatiotemporal collaboration module; a multi-twin agent reinforcement learning model construction module based on spatiotemporal cognitive differences; a multi-twin agent reinforcement learning model construction module based on spatiotemporal consistency representation; a graph convolution multi-twin agent reinforcement learning model construction module based on spatiotemporal dynamic collaboration graph. The above method and metaverse combine digital twin technology with artificial intelligence technology, and propose twin agents, which have all the characteristics of digital twins and all the characteristics of agents, so that multi-agent technology can be seamlessly integrated into digital twin systems and metaverses such as unmanned factories; combine the spatiotemporal information of multiple twin agents for more effective communication, and improve the global cognition and collaborative decision-making capabilities of multiple twin agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1A module diagram of a twin agent collaborative system provided by an embodiment of the present invention;

[0033] Figure 2 A module diagram of a twin agent collaborative system provided by an embodiment of the present invention;

[0034] Figure 3 A module diagram of a twin agent collaborative system provided by an embodiment of the present invention;

[0035] Figure 4 A module diagram of a twin agent collaborative system provided by an embodiment of the present invention;

[0036] Figure 5 An overall structural diagram provided for an embodiment of the present invention;

[0037] Figure 6 A diagram of a spatiotemporal consistency representation implementation process provided for an embodiment of the present invention;

[0038] Figure 7 An example diagram of a spatiotemporal cognitive coding network training process provided by an embodiment of the present invention;

[0039] Figure 8 An overall framework diagram provided for an embodiment of the present invention;

[0040] Fig. 9 A composition diagram of twin agent observation values ​​provided for an embodiment of the present invention;

[0041] Fig.10 A diagram of a spatiotemporal consistency representation implementation process provided for an embodiment of the present invention;

[0042] Fig.11 A schematic diagram of a GCN module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention are described in detail below in conjunction with the embodiments of the present invention.

[0044] 1. Basic Embodiments of the Invention

[0045] In the first aspect, an embodiment of the present invention provides a twin intelligent agent collaboration method, the method comprising: a preset scene acquisition step; a twin intelligent agent construction step; a twin intelligent agent spatiotemporal collaboration step. Technical effect: By combining digital twin technology with artificial intelligence technology, a twin intelligent agent is proposed, which has all the characteristics of digital twins and all the characteristics of intelligent agents, so that multi-agent technology can be seamlessly integrated into digital twin systems and metaverses such as unmanned factories; combining the spatiotemporal information of multiple twin intelligent agents for more effective communication, and improving the global cognition and collaborative decision-making capabilities of multiple twin intelligent agents.

[0046] Preferably, the method further includes: a step of constructing a multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences. Technical effect: By using the attention mechanism to aggregate different information between the twin agent and other twin agents in the relevant time and space, the twin agent is promoted to have a more comprehensive understanding of the environment, so that it can make correct behavioral collaborative decisions under the overall goal of collaborative decision-making.

[0047] Preferably, the method further includes: a step of constructing a multi-twin agent reinforcement learning model based on spatiotemporal consistency representation. Technical effect: The globality of the cognition of the twin agents is judged according to the size of the similarity, and then the twin agents with a global perspective are rewarded, because the twin agents with a global perspective should have a greater voice in collaborative decision-making, thereby improving the effect of collaborative decision-making. At the same time, the global cognitive ability of all twin agents can be gradually improved through the reward mechanism, thereby strengthening the cooperation of multiple twin agents.

[0048] Preferably, the method further includes: a step of constructing a graph convolution multi-twin agent reinforcement learning model based on a spatiotemporal dynamic collaborative graph. Technical effect: The spatiotemporal dynamic collaborative graph is constructed using the spatiotemporal information and feature information of the twin agents to capture the interaction relationship between the twin agents, filter out agents that do not need to interact, and strengthen the interaction between twin agents with large spatiotemporal correlation; then the spatiotemporal dynamic collaborative graph is used to perform spatiotemporal graph convolution to achieve the fusion of information from different twin agents, thereby increasing the perception field and global cognition of the twin agents, enabling them to make correct behavioral decisions.

[0049] In a second aspect, an embodiment of the present invention provides a twin intelligent collaborative system, such as Figure 1 The system includes: a preset scene acquisition module; a twin agent construction module; and a twin agent spatiotemporal collaboration module.

[0050] Preferably, if Figure 2 , the system also includes: a multi-twin agent reinforcement learning model construction module based on spatiotemporal cognitive differences.

[0051] Preferably, if Figure 3 , the system also includes: a multi-twin agent reinforcement learning model construction module based on spatiotemporal consistency representation.

[0052] Preferably, if Figure 4 , the system also includes: a graph convolution multi-twin agent reinforcement learning model construction module based on spatiotemporal dynamic collaborative graph.

[0053] In the third aspect, an embodiment of the present invention provides a multi-twin intelligent agent collaborative digital twin device, which includes a module of any one of the modules described in the second aspect embodiment.

[0054] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect.

[0055] In the fifth aspect, an embodiment of the present invention provides a multi-twin intelligent collaborative robot system, comprising a memory, a processor, and an artificial intelligence robot program stored in the memory and executable on the processor, characterized in that when the processor executes the program, the steps of the method described in any one of the embodiments of the first aspect are implemented.

[0056] In the sixth aspect, an embodiment of the present invention provides a multi-twin intelligent collaborative metaverse, including a metaverse, characterized in that the metaverse implements the steps of the method described in any one of the embodiments of the first aspect.

[0057] In one embodiment,

[0058] Preset scene acquisition step: Acquire the preset scene, for example, the preset scene is an unmanned factory, a disaster relief site, or a teaching site;

[0059] Steps for constructing a twin agent of a preset scene object: construct a twin agent based on the object of the preset scene; the twin agent (also called a digital twin agent) has all the characteristics of an agent and all the characteristics of a digital twin;

[0060] Twin agent spatiotemporal collaboration steps: collect data of each object in a preset scene through the Internet of Things (the data includes observation and / or operation status and / or results, etc.), as the data of the twin agent of each object, and input the data into a preset deep reinforcement learning model; use the preset deep reinforcement learning model to perform spatiotemporal communication and collaborative decision-making between the twin agents of each object in the preset scene; the preset deep reinforcement learning model includes a multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences or / and a multi-twin agent reinforcement learning model represented by spatiotemporal consistency or / and a graph convolution multi-twin agent reinforcement learning model based on a spatiotemporal dynamic collaboration graph.

[0061] Steps for constructing twin agents of preset scene objects: create a twin agent of each class of objects in the preset scene; instantiate the twin agent of each object in the preset scene; the twin agent of the object inherits the properties and methods of the twin agent of the class of objects to which the object belongs; and supplement the personalized properties and methods of the object;

[0062] The steps of constructing a multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences are as follows: the local observation values ​​of all twin agents in the preset scene are converted into an observation value cognitive matrix e through a neural network; e is an n×d-dimensional encoding matrix, where n is the number of twin agents, d is the size of the encoding dimension, and e represents the result of encoding the observation values ​​of all twin agents; the observation methods of all twin agents in the preset scene are converted into a reference cognitive matrix e' through a neural network; the observation methods of twin agents include the time of local observation, the spatial position of local observation, and the method of local observation; e' is an n×d'-dimensional encoding matrix, where n is the number of twin agents, d' is the size of the observation method dimension, and e' represents the result of encoding the observation methods of all twin agents; the observation value cognitive query matrix Q, the observation value cognitive value matrix K, and the observation value cognitive value matrix V are set to be the same as the observation value cognitive matrix e; the difference between the unit matrix and the dot product of the matrices Q' and K' is calculated, and added to the dot product of the matrices Q and K; the result is normalized using the soft max operation to obtain the attention observation cognitive coefficient matrix Where dk is used to scale the calculated results; the attention observation cognitive coefficient matrix a and the observation value cognitive value matrix V are used to calculate the attention observation cognitive matrix Use the observation value cognitive matrix to subtract the attention observation cognitive matrix, and then fuse it with the observation value cognitive matrix e converted by the neural network to generate a new input enew = Matrices Q' and K' are the inverse matrices of matrices Q and K;

[0063] The steps of building a multi-twin agent reinforcement learning model based on spatiotemporal consistency representation are as follows: a spatiotemporal cognitive coding network is created, including a multi-layer neural network, and multiple twin agents share the parameters of the spatiotemporal cognitive coding network; the spatiotemporal cognitive coding network is trained by taking the hidden state hi of the current twin agent as input and the hidden states hj of other twin agents as targets. In this process, the hidden state is the output of the Gated Recurrent Unit (GRU) layer of the twin agent policy network; the loss function of the spatiotemporal cognitive coding network is Among them, ψ is the parameter of the spatiotemporal cognitive coding network, N is the number of twin agents, hj i rep is the output of the spatiotemporal cognitive coding network, which represents the cognition of twin agent j converted into the spatiotemporal consistency representation of twin agent i; j represents a twin agent different from twin agent i; the similarity of the spatiotemporal consistency representation of the hidden state of each twin agent and the hidden state of other twin agents is used to represent its degree of cognition of the preset scene, as shown in the formula: Where i ranges from 1 to N, and j includes 0;

[0064] Therefore, the degree of cognition of all twin agents to the environment can be represented by the matrix W:

[0065] Use W to calculate the individual reward of each twin agent; the larger the W value, the higher the reward given to the twin agent.

[0066] The steps for constructing a graph convolution multi-twin agent reinforcement learning model based on spatiotemporal dynamic collaborative graph are as follows: the spatial information, temporal information, and content information of the twin agents are used to construct the temporal adjacency matrix M1, the spatial adjacency matrix M2, and the content adjacency matrix M3. The three adjacency matrices are intersected to obtain a comprehensive adjacency matrix M; X is used to represent the observation value encoding matrix of the twin agents, which is an n×d-dimensional matrix, where n is the number of twin agents, d is the size of the twin agent encoding dimension, and Q and K are the query matrix and key value matrix that are the same as X; the similarity between the twin agent observation value encodings is calculated through the self-attention mechanism Where dk is a normalization factor that scales the result. X' is used to represent the observation encoding matrix of the twin agents, which is an n×d'-dimensional matrix, where n is the number of twin agents, d' is the size of the twin agent observation encoding dimension, and Q' and K' are the same query matrix and key value matrix as X'. The similarity between the twin agent observation encodings is calculated through the self-attention mechanism. Where d'k is a normalization factor to scale the result; the obtained adjacency matrix M is used to filter out the similarity values ​​between twin agents that do not need to interact As the spatiotemporal interaction relationship between twin agents; the spatiotemporal interaction relationship between twin agents is used as an adjacency matrix for graph convolution to achieve information fusion between twin agents Where X (l) is the feature matrix of convolutional layer l, W (l) is a d×d dimensional trainable matrix of the lth convolutional layer, and σ is a nonlinear activation function.

[0067] In one embodiment,

[0068] Unmanned factories require the use of digital twin technology, so this application combines digital twin technology with artificial intelligence technology and proposes a twin intelligent agent, which has all the characteristics of digital twins and all the characteristics of intelligent agents, so that multi-agent technology can be seamlessly integrated into digital twin systems and metaverses such as unmanned factories.

[0069] Only by combining the spatiotemporal information of multiple twin intelligent agents can more effective communication be carried out and the global cognition and collaborative decision-making capabilities of multiple twin intelligent agents be improved, because spatiotemporal information can play an important role in the cognition and collaborative decision-making process of multiple twin intelligent agents.

[0070] First, in response to the problem of how to combine spatiotemporal information to achieve effective communication between twin agents to promote collaborative decision-making, this application proposes a multi-twin agent reinforcement learning method based on spatiotemporal cognitive differences and spatiotemporal consistency representation. By using the attention mechanism to aggregate the different information between the twin agent and other twin agents in the relevant time and space, the twin agent is promoted to have a more comprehensive understanding of the environment, so that it can make correct behavioral collaborative decisions under the overall goal of collaborative decision-making. This application further designs a spatiotemporal cognitive encoding network to obtain the spatiotemporal consistency representation of each twin agent's cognition relative to other twin agents, and then calculates the similarity between the spatiotemporal consistency representation and the hidden state of other twin agents to guide the training of the twin agent strategy network. The principle is to judge the global size of the twin agent's cognition according to the size of the similarity, and then reward the twin agent with a global view, because the twin agent with a global view should have a greater voice in collaborative decision-making, thereby improving the effect of collaborative decision-making. At the same time, the reward mechanism can also gradually improve the global cognitive ability of all twin agents, thereby strengthening the cooperation of multiple twin agents.

[0071] Second, in order to solve the problem of how to combine spatiotemporal information to obtain the interaction relationship between twin agents and perform information fusion, this application proposes a graph convolution multi-twin agent reinforcement learning method based on spatiotemporal dynamic collaborative graph. The spatiotemporal information and feature information of the twin agents are used to construct a spatiotemporal dynamic collaborative graph to capture the interaction relationship between twin agents, filter out agents that do not need to interact, and strengthen the interaction between twin agents with large spatiotemporal correlation; then the spatiotemporal dynamic collaborative graph is used to perform spatiotemporal graph convolution to achieve the fusion of information from different twin agents, thereby increasing the perception field and global cognition of the twin agents, enabling them to make correct behavioral decisions.

[0072] This application objectifies the unmanned factory. There are many objects and their methods in the unmanned factory. The object categories in the unmanned factory include personnel, machines, raw materials, and environment. Each type of object includes many different individual objects. Objects have time and space dimensions when they are observed. For example, personnel have time and space dimensions when recording data, machines have time and space dimensions when collecting data through cameras and sensors, the environment also has time and space dimensions when collecting environmental data through monitors and sensors, and raw materials have time and space dimensions when collecting data through cameras and sensors. Object methods include observation methods and operation methods. The observation method is to observe the status and results of the operation. The observation data obtained by different observation methods will be different. For example, observation methods with different accuracies will obtain observation results with different accuracies.

[0073] This application constructs twin intelligent bodies based on objects in unmanned factories. For each class of objects in the unmanned factory, a twin intelligent body of the class of objects is created, and then the twin intelligent body of the object is instantiated for each object. The twin intelligent body of the object inherits the properties and methods of the twin intelligent body of the class of objects to which the object belongs and supplements the personalized properties and methods of the object. Twin intelligent body is an intelligent body that has all the characteristics of an intelligent body, and at the same time, the twin intelligent body has all the characteristics of a digital twin. Because the twin intelligent body is an intelligent body, the intelligent twin of an object has intelligence, can make autonomous decisions through an artificial intelligence model (deep learning model or knowledge base reasoning machine), has both attributes and methods, can operate and behave autonomously, and the intelligent twins of different objects are relatively independent and need to communicate and collaborate through communication. Because the twin intelligent body is a digital twin, any information of an object can be obtained through the intelligent twin of the object, the object can be controlled to execute any method of the object, and any behavior of the object can be simulated through the digital twin of an object, and the result of the behavior can be predicted.

[0074] This application uses a deep reinforcement learning model to make collaborative decisions for the twin agents of objects in unmanned factories. The twin agents of each object observe and operate independently, and the status and results of the operation are input into the deep reinforcement learning model through observation data, and then efficient collaborative decision-making is achieved through spatiotemporal communication between the twin agents of each object.

[0075] The scheme of the specific embodiment:

[0076] 1. Spatiotemporal consistency representation Multi-twin agent reinforcement learning method based on spatiotemporal cognitive differences and spatiotemporal consistency representation

[0077] The algorithm proposed in this application is mainly divided into a spatiotemporal cognitive difference module, a spatiotemporal consistency representation module and an overall algorithm. Figure 5 The structure diagram of the overall algorithm is shown, in which the upper middle part of the structure diagram represents the overall process of the algorithm; the lower middle part of the structure diagram is the detailed structure of the twin agent strategy network; the CD on the left represents the spatiotemporal cognitive difference module, and the CR on the right represents the spatiotemporal consistency representation module.

[0078] The detailed implementation process of the spatiotemporal cognitive difference module is as follows: Figure 5As shown, the specific implementation steps are as follows: first, the local observation values ​​of all twin agents are converted into an observation value cognitive matrix e through a fully connected layer FC. e is an n×d-dimensional encoding matrix, where n is the number of twin agents and d is the size of the encoding dimension, which represents the result of encoding the observation values ​​of all twin agents. Then, the observation methods of all twin agents are converted into a reference cognitive matrix e' through a fully connected layer FC. The observation methods of twin agents include the time of local observation, the spatial position of local observation, and the method of local observation. e' is an n×d'-dimensional encoding matrix, where n is the number of twin agents and d' is the size of the observation method dimension, which represents the result of encoding the observation methods of all twin agents.

[0079] If the difference information between the twin agent and other twin agents is simply added up, the interaction relationship between the twin agents will not be reflected. Therefore, the present application adopts the following method to fuse the difference information of the twin agents. The present application first uses the self-attention mechanism to calculate the weighted cognition of all twin agents on the environment. This operation is called atten ion, in which the observation value cognition query matrix Q, the observation value cognition key matrix K and the observation value cognition value matrix V of the self-attention mechanism are set to be the same as the observation value cognition matrix e, and the dot product of the matrix Q and K is calculated; the observation mode cognition query matrix Q', the observation mode cognition key matrix K' and the observation mode cognition value matrix V' of the self-attention mechanism are set to be the same as the observation mode cognition matrix e', and the difference between the unit matrix and the dot product of the matrix Q' and K' is calculated, and added to the dot product of the matrix Q and K. The result is then normalized using a soft max operation. The following attention observation cognition coefficient matrix can be generated through the above calculations, as shown in the following formula:

[0080]

[0081] Where dk is used to scale the calculated result. Then the attention observation cognitive matrix is ​​calculated with the observation value cognitive value matrix V, as shown in the following formula:

[0082]

[0083] On the basis of the attention observation cognitive matrix, the following operations are used to obtain the cognitive differences between the twin agents, so as to evaluate the importance of different twin agents and integrate the difference information. First, the attention observation cognitive matrix is ​​subtracted from the cognitive matrix of the twin agents. The essence is that by subtracting the similar parts, the different parts can be obtained naturally, and the attention cognitive difference matrix of the twin agents is obtained. Then, the information is fused by addition to generate a new input enew. Therefore, all twin agents can receive cognitive difference information from other twin agents, thereby completing the communication between twin agents. This implementation process is shown in the following formula:

[0084]

[0085] Therefore, through the above operation, the twin intelligent agent retains its own local observation value in the environment while integrating the information of attention cognition difference, and takes into account the influence of the difference in observation method on the observation value, thereby eliminating the influence of the reasonable difference caused by the observation method in the difference in observation value. The results observed by different twin intelligent agents at different times and spaces using different observation methods are naturally different. This difference is objective and reasonable. For example, if the spatial distance between the two twin intelligent agents is relatively far, the natural similarity of the observation value is relatively small. Therefore, when considering the difference in observation value, it is necessary to use the difference in observation method as a reference, and then regard the difference caused by the observation method as a normal difference, and strive to eliminate the difference caused by improper operation, observation error, interference, etc., so that the corrected local observation is more accurate. This method can enable the twin intelligent agent to have a better understanding of the environment. Each twin intelligent agent adjusts its own understanding of the environment according to the understanding of the environment of other twin intelligent agents, so that the twin intelligent agent can make correct behavioral collaborative decisions.

[0086] Temporal and spatial consistency representation:

[0087] Based on the spatiotemporal cognitive difference strategy, communication between twin agents is realized, that is, how to integrate useful information to help twin agents make collaborative behavioral decisions in the environment. At the same time, when integrating information from other twin agents, the objective differences in observation time, space, and methods are taken into account, and the information from other twin agents is used to help their own decision-making while seeking common ground while reserving differences. However, this difference still exists objectively and cannot be completely eliminated. It is impossible to make the cognition of each twin agent completely consistent with the global cognition. Then, in another way of thinking, if local cognition close to global cognition can be found, it is also very meaningful to let the twin agents with local cognition close to global cognition have a greater voice in collaborative decision-making. This is equivalent to improving the globality of collaborative decision-making. Therefore, after using the spatiotemporal cognitive difference strategy based on the attention mechanism to achieve effective communication to eliminate unreasonable differences, in order to further improve the global awareness of twin agents, this application proposes an indirect communication method to achieve the consistent spatiotemporal cognition of the twin agents to the environment, thereby strengthening the cooperation of twin agents.

[0088] Due to individual differences, when facing the same environment, the global cognition obtained by each twin agent in the previous step is not exactly the same, because there are objective differences in the time, space, and methods of observation of different twin agents, and such differences cannot be reached through communication. If the global cognition of all twin agents can be consistent, they can achieve better cooperation. Therefore, this application proposes a spatiotemporal consistency representation strategy, such as Figure 6 The detailed implementation process is as follows: First, a spatiotemporal cognitive encoding network is trained to obtain the spatiotemporal consistency representation of each twin agent. Secondly, the representation of the twin agent should be similar to the hidden state of other twin agents. The more similar, the better the twin agent's cognition of the environment, and the more the behavior of the twin agent should be encouraged.

[0089] This application first creates a spatiotemporal cognitive coding network, including two fully connected (FC) layers, and the twin agents share the parameters of the neural network. Figure 7As shown, the spatiotemporal cognitive coding network is trained by taking the hidden state hi of the current twin agent as input and the hidden state hj of other twin agents as the target. In this process, the hidden state is the output of the Gated Recurrent Unit (GRU) layer of the twin agent strategy network. This application refers to the output of the above network as the spatiotemporal consistency representation of the twin agent's cognition of the environment. This way of training the network can also be regarded as an indirect means of communication. The reason is that there is no explicit information transmission when the twin agents make collaborative behavioral decisions in this process. It is only through this process that the global cognitive ability of each twin agent is measured, and then the environmental reward value distribution ratio of different twin agents in the twin agent's strategy network is adjusted according to the level of the global cognitive ability of each twin agent, thereby indirectly improving the global cognitive ability and collaborative decision-making ability of all twin agents through the reinforcement process. The cognitive coding network makes full use of the prior knowledge of observation time, space and method encoding to speed up the speed and efficiency of training convergence. The loss function of the spatiotemporal cognitive coding network is calculated as follows:

[0090]

[0091] Among them, ψ is the parameter of the spatiotemporal cognitive coding network, N is the number of twin agents, hi is the hidden state of twin agent i, h ii rep, hjj rep, and hj i rep are the outputs of the spatiotemporal cognitive coding network, h ii rep is the output of the spatiotemporal cognitive coding network that converts the cognition of twin agent i into the spatiotemporal representation of twin agent i, hj i rep is the output of the spatiotemporal cognitive coding network that converts the cognition of twin agent j into the spatiotemporal representation of twin agent i, and hjj rep is the output of the spatiotemporal cognitive coding network that converts the cognition of twin agent j into the spatiotemporal representation of twin agent j. It indicates that the cognition of twin agent j is converted into the spatiotemporal consistency representation of twin agent i, thereby converting the cognition of other twin agents into the spatiotemporal consistency representation with the current twin agent, and then measuring the consistency of these local cognitions. If the similarity between the local cognition of other twin agents and the local cognition of the current twin agent is higher, it means that the local cognition of the current twin agent is closer to the global cognition. j represents a twin agent that is different from the twin agent i.

[0092] This application uses the similarity of the temporal and spatial consistency representation of the hidden state of each twin agent and the hidden state of other twin agents to represent its degree of cognition of the environment, as shown in the following formula (where i ranges from 1 to N and j includes 0):

[0093] Therefore, the degree of cognition of all twin agents to the environment can be represented by the matrix W:

[0094]

[0095] Finally, the behavior of the twin agents is adjusted by updating the strategy network of the twin agents to achieve the consistent spatiotemporal cognition of the twin agents to the environment. Specifically, W is used to calculate the individual reward of the twin agents. The larger the value, the better the twin agents have cognition of the environment, and the higher the reward should be. Therefore, the matrix W is multiplied by the global reward as the individual reward of the twin agents.

[0096] The above is the implementation process of the spatiotemporal consistency representation module proposed in this application. It guides the training of the twin intelligent agent strategy network through indirect communication, promotes the twin intelligent agent's consistent spatiotemporal cognition of the environment, and then strengthens the cooperation of the twin intelligent agents.

[0097] The essence of the spatiotemporal cognitive coding network is to perform spatiotemporal conversion on the observation values ​​of each twin agent, so that they can be compared in the same spatiotemporal reference system. Otherwise, if the comparison is made directly, it is likely that the similarity is very low due to the opposite observation angles of the two twin agents. If they are adjusted to the same spatiotemporal reference system, the similarity is actually very high. This is the main role played by the spatiotemporal cognitive coding network. The spatiotemporal cognitive coding network does not convert local cognition into global cognition, because there is no known global cognition for training, so a network with such functions cannot be trained at all. If the global cognition close to each local cognition is used as the training standard for global cognition, it is also difficult to implement, mainly because the spatiotemporal reference system of global cognition is unknown and difficult to define, and it is difficult to unify the local cognition of each twin agent into the spatiotemporal reference system of global cognition. Therefore, the spatiotemporal cognitive coding network of this application is only used for spatiotemporal cognitive coding of local cognition, not for generating global cognition. However, global cognition is crucial to the decision-making of the strategy network of twin agents, and how to generate global cognition is the next issue to be studied.

[0098] 2. Graph convolution multi-twin agent reinforcement learning method based on spatiotemporal dynamic collaborative graph

[0099] The multi-twin agent reinforcement learning method based on the representation of spatiotemporal cognitive differences and spatiotemporal consistency reduces unreasonable cognitive differences by integrating the attention cognitive difference matrix of twin agents in the strategy network of twin agents, which plays a good role in eliminating the difficulties brought to the training and testing of the strategy network of twin agents by the contradictions between the local cognitions of different twin agents, and can improve the collaborative decision-making ability and effect of the strategy network of twin agents; at the same time, the global cognitive ability of each twin agent is evaluated through the spatiotemporal consistency representation and spatiotemporal cognitive encoding network, and the global cognitive ability of twin agents with strong global cognitive ability is gradually improved by rewarding twin agents, thereby improving the collaborative decision-making ability and effect of twin agents.

[0100] But in essence, the multi-twin agent reinforcement learning method based on spatiotemporal cognitive differences and spatiotemporal consistency representations has not been able to transform the local cognition of the intelligent twins into global cognition. How can the local cognition of the intelligent twins be transformed into global cognition? This requires each twin agent to abandon the observation time, space and method of the small self, and observe from the perspective of the big self, which is commonly known as the "God's perspective" or the global perspective. How can the cognition obtained from the local perspective of each agent be elevated to global cognition? This requires the use of deep learning models to obtain global cognition based on local cognitive fusion and prediction.

[0101] The graph convolution multi-twin agent reinforcement learning method based on spatiotemporal dynamic collaborative graph proposed in this application is mainly divided into a spatiotemporal dynamic collaborative graph module and a spatiotemporal graph convolution module.

[0102] Figure 8 The overall framework is presented, where DGC represents the spatiotemporal dynamic collaborative graph module and GC represents the spatiotemporal graph convolution module.

[0103] Spatiotemporal dynamic collaborative graph module:

[0104] The detailed implementation process of the spatiotemporal dynamic collaborative graph module is as follows. In the first stage, features are extracted from the observation values ​​of the twin intelligent agents based on prior domain knowledge, and an adjacency matrix is ​​constructed according to its observation range. When the twin intelligent agent can observe the cooperative object or the competitive object, its corresponding position is the feature of the cooperative object or the competitive object, otherwise it is 0. Observables are divided into temporal observables, spatial observables, and content observables; for example, an object at a point in time can observe objects at past time points, but cannot observe objects at future time points; an object in space can observe unobstructed objects within its perception range (visual, auditory, or tactile, etc.), but cannot observe objects outside its perception range, nor can it observe occluded objects; an object can observe other objects within the range of its observed content type, but cannot observe other objects outside the range of its observed content type. The detailed composition of the twin intelligent agent observation values ​​is as follows: Fig. 9 shown.

[0105] Therefore, by Fig. 9 As shown, when the Vi sib le eigenvalue of a twin intelligent body of a cooperative object corresponding to the twin intelligent body's observation value is 1, it means that the twin intelligent body of this cooperative object can be observed; when the Vi s ib le eigenvalue of a twin intelligent body of a competitive object corresponding to the twin intelligent body's observation value is -1, it means that the twin intelligent body of this competitive object can be observed. The reason why the eigenvalue between competitive objects is -1 is that the communication and interaction between competitive objects should be reduced as much as possible to avoid conflicts and deadlocks during decision-making, but communication cannot be completely eliminated, because conflicts between competitive objects are objective existences and should be taken into account when making collaborative decisions; when this value is 0, it means that the twin intelligent body cannot be observed. Therefore, the adjacency matrix constructed for the twin intelligent body in this application is as follows. Taking the scenario of 5 twin intelligent bodies as an example, Fig.10 shown.

[0106] This adjacency matrix indicates that twin agent 1 can observe twin agent 3, but twin agent 3 cannot observe twin agent 1; twin agent 4 can observe twin agent 2, and twin agent 2 can also observe twin agent 4. Although in this example, 0 is used to indicate no interaction and 1 is used to indicate interaction, in fact, the intensity of interaction can also be represented by numbers from -1 to 1. The stronger the interaction, the closer it is to 1; the stronger the mutual exclusivity, the closer it is to -1; and the weaker the interaction, the closer it is to 0.

[0107] The adjacency matrix includes the time adjacency matrix M1, the space adjacency matrix M2, and the content adjacency matrix M3. The three adjacency matrices are intersected to obtain a comprehensive adjacency matrix M.

[0108] In the first stage, this application uses the spatial information, temporal information, and content information of the twin agents to construct an adjacency matrix (using expert knowledge, such as rules in terms of temporal and spatial content), and further uses the feature information of the twin agents in the second stage to capture the interactive relationship between the twin agents (automatically extracting correlations through data and self-attention mechanisms). First, the similarity between the twin agent observation value encodings is calculated through the self-attention mechanism. Here, X represents the twin agent observation value encoding matrix, which is an n×d-dimensional matrix, where n is the number of twin agents, d is the size of the twin agent encoding dimension, and Q and K are the same query matrix and key value matrix as X:

[0109]

[0110] Where dk is a normalization factor that scales the result.

[0111] Then, the similarity between the twin agent observation encodings is calculated through the self-attention mechanism. Here, X' is used to represent the twin agent observation encoding matrix, which is an n×d'-dimensional matrix, where n is the number of twin agents, d' is the size of the twin agent observation encoding dimension, and Q' and K' are the same query matrix and key value matrix as X':

[0112]

[0113] Where d'k is a normalization factor that scales the result.

[0114] The above formula can be used to calculate the similarity between the observation encodings of any two twin agents and the similarity between the observation methods. Then, the obtained adjacency matrix M is used to filter out the similarity between the twin agents that do not need to interact. At the same time, the smaller the similarity, the less interaction is needed. For example, the greater the spatiotemporal similarity, the more interaction is needed. The greater the similarity of the observations, the more interaction is needed:

[0115] Finally, the matrix M is normalized by the softmax function. For two twin agents that do not interact, it is represented as 0 in the adjacency matrix M, indicating that they do not need to interact. For two twin agents that interact, it is represented as a number greater than or equal to -1 and less than or equal to 1 and not equal to 0 in the adjacency matrix M. If the normalization operation is performed directly, due to the characteristics of the softmax function itself, an edge weight will be assigned between the two twin agents that do not need to interact, which is unreasonable. Therefore, this application uses an extremely small value -10e10 to replace 0 in the formula. In this way, the calculated edge weight of the dynamic collaborative graph is 0, that is, the information of the twin agent is not introduced when performing information fusion:

[0116]

[0117] Next, perform the normalization operation:

[0118]

[0119] where e· represents the exponential operation, and the right side of the formula represents the interaction relationship between twin agents i and j before normalization, It represents the sum of the exponential operation of the interaction relationship between twin agent i and any twin agent k; the left side of the formula calculates Represents the interaction relationship between twin agents i and j after normalization. When its value is 0, twin agent i cannot obtain information about twin agent j. When its value is not 0, it is a number greater than 0 and less than or equal to 1, and twin agent i can obtain information about twin agent j. When its value is large, it indicates that the greater the correlation, the more interactions are needed; when its value is small, it indicates that the smaller the correlation, the less interactions are needed; when its value is very small, it indicates that they may be in a competitive relationship and their interactions need to be minimized. Therefore, the dynamic collaborative graph M obtained by the above operation can adapt to the dynamic changes of the interaction relationship between twin agents. When there is no interaction between twin agents, no communication is performed between them; and when there is interaction between twin agents, soft edge weights are used to introduce the influence of twin agent feature information on their interaction.

[0120] Spatiotemporal graph convolution module:

[0121] After obtaining the interaction relationship between the twin agents, a suitable method is needed to fuse the information of the twin agents. The original GCN (Graph Convolutional Network) method only uses the adjacency matrix to fuse node feature information, without considering the influence of the node's own information, let alone the influence of different observation methods such as time and space of the node. Therefore, this application uses an improved graph convolution, and uses the obtained spatiotemporal interaction relationship between the twin agents as the adjacency matrix for graph convolution to achieve information fusion between the twin agents. One convolution layer is:

[0122]

[0123] Dynamic collaboration diagram So D can be transformed into a unit matrix, and since the value on the diagonal of M is greater than 0, the conditions for using GCN are met, and a convolutional layer can be simplified as:

[0124] Where X (l) is the feature matrix of convolutional layer l, W (l) is a d×d dimensional trainable matrix of the lth convolutional layer, σ is a nonlinear activation function, and this application uses a graph convolution structure with more than two layers, such as Fig.11 As shown. Finally, the information output by multi-layer (for example, two-layer) graph convolution is X(N) (for example, X(2)), which realizes the communication between twin agents. In this way, the twin agent integrates the information of other twin agents interacting with it, and takes into account the coordination necessity and interaction strength of different interactions during interaction. For example, the more similar the observation values ​​are and the closer the time and space are, the greater the coordination necessity and interaction strength are, which enables the twin agents to make more efficient behavioral coordination decisions.

[0125] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

Claims

1. A twin agent collaboration method, characterized in that: The method comprises: Preset scene acquisition steps: acquire the preset scene; Steps for building a twin intelligent agent: build a twin intelligent agent based on the objects in the preset scene; the twin intelligent agent has the characteristics of an intelligent agent and also has the characteristics of a digital twin; Twin agent spatiotemporal collaboration step: collect data of each object in a preset scene through the Internet of Things as the data of the twin agent of each object, and input the data into a preset deep reinforcement learning model; use the preset deep reinforcement learning model to perform spatiotemporal communication and collaborative decision-making between the twin agents of each object in the preset scene; the preset deep reinforcement learning model is a multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences or / and a multi-twin agent reinforcement learning model represented by spatiotemporal consistency or / and a graph convolution multi-twin agent reinforcement learning model based on a spatiotemporal dynamic collaboration graph; The multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences uses an attention mechanism to aggregate different information between twin agents and other twin agents in related space and time; the multi-twin agent reinforcement learning model characterized by spatiotemporal consistency judges the global size of twin agent cognition according to the size of similarity, and then rewards twin agents with a global perspective; the graph convolution multi-twin agent reinforcement learning model based on spatiotemporal dynamic collaborative graph uses the spatiotemporal information and feature information of twin agents to construct a spatiotemporal dynamic collaborative graph to capture the interaction relationship between twin agents, filter out agents that do not need to interact, and strengthen the interaction between twin agents with large spatiotemporal correlation; then the spatiotemporal dynamic collaborative graph is used to perform spatiotemporal graph convolution to achieve the fusion of information from different twin agents; The steps of constructing a multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences are as follows: the local observation values ​​of all twin agents in the preset scene are converted into an observation value cognitive matrix e through a neural network; e is an n×d-dimensional encoding matrix, where n is the number of twin agents, d is the size of the encoding dimension, and e represents the result of encoding the observation values ​​of all twin agents; the observation methods of all twin agents in the preset scene are converted into a reference cognitive matrix e' through a neural network; the observation methods of twin agents include the time of local observation, the spatial position of local observation, and the method of local observation; e' is an n×d'-dimensional encoding matrix, where n is the number of twin agents, d' is the size of the observation method dimension, and e' represents the result of encoding the observation methods of all twin agents; the observation value cognitive query matrix Q, the observation value cognitive value matrix K, and the observation value cognitive value matrix V are set to be the same as the observation value cognitive matrix e; the difference between the unit matrix and the dot product of the matrices Q' and K' is calculated, and added to the dot product of the matrices Q and K; the result is normalized using a soft max operation to obtain the attention observation cognitive coefficient matrix Where dk is used to scale the calculated results; the attention observation cognitive coefficient matrix a and the observation value cognitive value matrix V are used to calculate the attention observation cognitive matrix Use the observation value cognitive matrix to subtract the attention observation cognitive matrix, and then fuse the information with the observation value cognitive matrix e converted by the neural network to generate a new input The steps of building a multi-twin agent reinforcement learning model based on spatiotemporal consistency representation are as follows: a spatiotemporal cognitive coding network is created, and multiple twin agents share the parameters of the spatiotemporal cognitive coding network; the spatiotemporal cognitive coding network is trained by taking the hidden state hi of the current twin agent as input and the hidden states hj of other twin agents as targets; the loss function of the spatiotemporal cognitive coding network is Among them, ψ is the parameter of the spatiotemporal cognitive coding network, N is the number of twin agents, hji rep is the output of the spatiotemporal cognitive coding network, which indicates that the cognition of twin agent j is converted into the spatiotemporal consistency representation of twin agent i; j represents a twin agent different from twin agent i; the similarity of the spatiotemporal consistency representation of the hidden state of each twin agent and the hidden state of other twin agents is used to represent its degree of cognition of the preset scene. Where i ranges from 1 to N, and j includes 0; the degree of cognition of all twin agents to the environment can be represented by the matrix W; Use W to calculate the individual reward of each twin agent; the larger the W value, the higher the reward given to the twin agent; The steps for constructing a graph convolution multi-twin agent reinforcement learning model based on spatiotemporal dynamic collaborative graph are as follows: the spatial information, temporal information, and content information of the twin agents are used to construct the temporal adjacency matrix M1, the spatial adjacency matrix M2, and the content adjacency matrix M3. The three adjacency matrices are intersected to obtain a comprehensive adjacency matrix M; X is used to represent the observation value encoding matrix of the twin agents, which is an n×d-dimensional matrix, where n is the number of twin agents, d is the size of the twin agent encoding dimension, and Q and K are the query matrix and key value matrix that are the same as X; the similarity between the twin agent observation value encodings is calculated through the self-attention mechanism Where dk is a normalization factor that scales the result. X' is used to represent the observation encoding matrix of the twin agents, which is an n×d'-dimensional matrix, where n is the number of twin agents, d' is the size of the twin agent observation encoding dimension, and Q' and K' are the same query matrix and key value matrix as X'. The similarity between the twin agent observation encodings is calculated through the self-attention mechanism. Where d'k is a normalization factor to scale the result; the obtained adjacency matrix M is used to filter out the similarity values ​​between twin agents that do not need to interact As the spatiotemporal interaction relationship between twin agents; the spatiotemporal interaction relationship between twin agents is used as an adjacency matrix for graph convolution to achieve information fusion between twin agents Where X (l) is the feature matrix of convolutional layer l, W (l) is a d×d dimensional trainable matrix of the lth convolutional layer, and σ is a nonlinear activation function.

2. A twin agent collaborative system, characterized in that: The system comprises: Preset scene acquisition module: obtain preset scenes; Twin intelligent agent construction module: builds a twin intelligent agent based on the objects in the preset scene; the twin intelligent agent has the characteristics of an intelligent agent and also has the characteristics of a digital twin; Twin agent spatiotemporal collaboration module: collect data of each object in a preset scene through the Internet of Things, as the data of the twin agent of each object, and input the data into a preset deep reinforcement learning model; use the preset deep reinforcement learning model to perform spatiotemporal communication and collaborative decision-making between the twin agents of each object in the preset scene; the preset deep reinforcement learning model is a multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences or / and a multi-twin agent reinforcement learning model represented by spatiotemporal consistency or / and a graph convolution multi-twin agent reinforcement learning model based on a spatiotemporal dynamic collaboration graph; The multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences uses an attention mechanism to aggregate different information between twin agents and other twin agents in related space and time; the multi-twin agent reinforcement learning model characterized by spatiotemporal consistency judges the global size of twin agent cognition according to the size of similarity, and then rewards twin agents with a global perspective; the graph convolution multi-twin agent reinforcement learning model based on spatiotemporal dynamic collaborative graph uses the spatiotemporal information and feature information of twin agents to construct a spatiotemporal dynamic collaborative graph to capture the interaction relationship between twin agents, filter out agents that do not need to interact, and strengthen the interaction between twin agents with large spatiotemporal correlation; then the spatiotemporal dynamic collaborative graph is used to perform spatiotemporal graph convolution to achieve the fusion of information from different twin agents; The building module of the multi-twin agent reinforcement learning model based on spatiotemporal cognitive differences: the local observation values ​​of all twin agents in the preset scene are converted into an observation value cognitive matrix e through a neural network; e is an n×d-dimensional encoding matrix, where n is the number of twin agents, d is the size of the encoding dimension, and e represents the result of the observation value encoding of all twin agents; the observation mode of all twin agents in the preset scene is converted into a reference cognitive matrix e' through a neural network; the observation mode of the twin agent includes the time of local observation, the spatial position of local observation, and the system of local observation; e' is an n×d'-dimensional encoding matrix, where n is the number of twin agents, d' is the size of the observation mode dimension, and e' represents the result of the observation mode encoding of all twin agents; the observation value cognitive query matrix Q, the observation value cognitive value matrix K and the observation value cognitive value matrix V are set to be the same as the observation value cognitive matrix e; the difference between the unit matrix and the dot product of the matrices Q' and K' is calculated, and added to the dot product of the matrices Q and K; the result is normalized using the soft max operation to obtain the attention observation cognitive coefficient matrix Where dk is used to scale the calculated results; the attention observation cognitive coefficient matrix a and the observation value cognitive value matrix V are used to calculate the attention observation cognitive matrix Use the observation value cognitive matrix to subtract the attention observation cognitive matrix, and then fuse the information with the observation value cognitive matrix e converted by the neural network to generate a new input The building module of multi-twin agent reinforcement learning model based on spatiotemporal consistency representation: a spatiotemporal cognitive coding network is created, and multiple twin agents share the parameters of the spatiotemporal cognitive coding network; the spatiotemporal cognitive coding network is trained by taking the hidden state hi of the current twin agent as input and the hidden states hj of other twin agents as targets; the loss function of the spatiotemporal cognitive coding network is Among them, ψ is the parameter of the spatiotemporal cognitive coding network, N is the number of twin agents, hji rep is the output of the spatiotemporal cognitive coding network, which indicates that the cognition of twin agent j is converted into the spatiotemporal consistency representation of twin agent i; j represents a twin agent different from twin agent i; the similarity of the spatiotemporal consistency representation of the hidden state of each twin agent and the hidden state of other twin agents is used to represent its degree of cognition of the preset scene. Where i ranges from 1 to N, and j includes 0; the degree of cognition of all twin agents to the environment can be represented by the matrix W; Use W to calculate the individual reward of each twin agent; the larger the W value, the higher the reward given to the twin agent; The construction module of the graph convolution multi-twin agent reinforcement learning model based on spatiotemporal dynamic collaborative graph: the spatial information, temporal information and content information of the twin agents are used to construct the temporal adjacency matrix M1, the spatial adjacency matrix M2 and the content adjacency matrix M3. The three adjacency matrices are intersected to obtain the comprehensive adjacency matrix M; X is used to represent the observation value encoding matrix of the twin agents, which is an n×d-dimensional matrix, where n is the number of twin agents, d is the size of the twin agent encoding dimension, and Q and K are the query matrix and key value matrix that are the same as X; the similarity between the twin agent observation value encodings is calculated through the self-attention mechanism Where dk is a normalization factor that scales the result. X' is used to represent the observation encoding matrix of the twin agents, which is an n×d'-dimensional matrix, where n is the number of twin agents, d' is the size of the twin agent observation encoding dimension, and Q' and K' are the same query matrix and key value matrix as X'. The similarity between the twin agent observation encodings is calculated through the self-attention mechanism. Where d'k is a normalization factor to scale the result; the obtained adjacency matrix M is used to filter out the similarity values ​​between twin agents that do not need to interact As the spatiotemporal interaction relationship between twin agents; the spatiotemporal interaction relationship between twin agents is used as an adjacency matrix for graph convolution to achieve information fusion between twin agents Where X (l) is the feature matrix of convolutional layer l, W (l) is a d×d dimensional trainable matrix of the lth convolutional layer, and σ is a nonlinear activation function.

3. A twin intelligent space-time collaborative metaverse, characterized by: The metaverse implements the steps of any method described in claim 1.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 are implemented.

Images