Method, device, equipment and storage medium for determining intelligent agent processing

Abstract

The present application discloses a method, device, equipment and storage medium for determining agent processing, which belongs to the field of artificial intelligence technology. The method includes: whenever a decision cycle is reached, determining the first environmental feature corresponding to the target agent; obtaining the second environmental feature corresponding to the neighbor agent of the target agent, the neighbor agent is an agent preset in the multi-agent system to communicate with the target agent; based on the first environmental feature and the second environmental feature, generating a target fusion environmental feature; inputting the target fusion environmental feature into the policy network to obtain the target processing that the target agent needs to perform in the current decision cycle. The use of this application can improve the efficiency of the agent in determining the processing that needs to be performed in the decision cycle.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for determining intelligent agent processing. Background Art

[0002] A multi-agent system is composed of multiple agents, and each agent can work together to complete a complex task. Each agent is in the same environment, and each agent can observe the environment around it and extract features from the observed environment to obtain corresponding environmental features. Each agent can calculate the obtained environmental features through a policy network to determine the processing that needs to be performed in each decision cycle. Among them, the agent can be an entity that makes independent decisions in a specified environment, and the entity can exist in the form of hardware, such as a drone, a robot, etc. It can also exist in the form of software, such as an artificial intelligence (AI) game character in an electronic game. The environment in which each corresponding agent is located can be a real environment, such as an image taken by a drone's camera. It can also be a virtual environment, such as information about other AI game characters within the attack range of an AI game character.

[0003] Since an agent observes a limited environment, the environmental features obtained include less information about the environment. Therefore, the processing that the agent needs to perform in the environment is determined based on only the environmental features with less information, resulting in inaccurate determination of the processing that the agent needs to perform. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and storage medium for determining agent processing, which can improve the efficiency of the agent in determining the processing to be executed during the decision cycle. The technical solution is as follows:

[0005] In a first aspect, a method for determining agent processing is provided, the method comprising:

[0006] Whenever a decision cycle is reached, determining a first environmental feature corresponding to the target agent;

[0007] Acquire a second environmental feature corresponding to a neighboring agent of the target agent, wherein the neighboring agent is an agent preset in the multi-agent system to communicate with the target agent;

[0008] Based on the first environmental feature and the second environmental feature, generating a target fusion environmental feature;

[0009] The target fusion environment features are input into the policy network to obtain the target processing that the target agent needs to perform in the current decision cycle.

[0010] Optionally, before obtaining the second environment feature corresponding to the neighboring agent of the target agent, the method further includes:

[0011] Based on the preset corresponding relationship between the target agent and the neighboring agents, the neighboring agents of the target agent are determined.

[0012] Optionally, generating a target fusion environment feature based on the first environment feature and the second environment feature includes:

[0013] Inputting the first environmental feature into an encoder to obtain a first feature to be fused;

[0014] Inputting the second environmental feature into the encoder to obtain a second feature to be fused;

[0015] The first feature to be fused and the second feature to be fused are fused to obtain a target fusion environment feature.

[0016] Optionally, generating a target fusion environment feature based on the first environment feature and the second environment feature includes:

[0017] Inputting the first environmental features into a plurality of encoders respectively to obtain a plurality of third features to be fused;

[0018] Inputting the second environmental features into the multiple encoders respectively to obtain multiple fourth features to be fused;

[0019] The third feature to be fused and the fourth feature to be fused obtained by the same encoder are respectively fused to obtain a plurality of fused features to be spliced;

[0020] The multiple fusion features to be spliced ​​are spliced ​​together to obtain target fusion environment features.

[0021] Optionally, generating a target fusion environment feature based on the first environment feature and the second environment feature includes:

[0022] Performing a fusion process on the first environmental feature and the second environmental feature to obtain a fusion environmental feature after a first fusion process;

[0023] Set n to 2, and obtain the fusion environment features obtained after the neighboring agents of the target agent perform the n-1th fusion process;

[0024] The fused environment features obtained after the n-1th fusion process of the neighboring intelligent bodies of the target intelligent body are fused with the fused environment features after the n-1th fusion process of the target intelligent body to obtain the fused environment features after the nth fusion process;

[0025] If n is less than N, add 1 to the value of n, and proceed to the step of obtaining the fusion environment features obtained after the n-1th fusion process of the neighboring agents of the target agent, where N is the total number of fusion processes preset, and N is greater than or equal to 2;

[0026] If n is equal to N, the fusion environment features obtained after each fusion process are spliced ​​to obtain the target fusion environment features.

[0027] In a second aspect, a device for determining agent processing is provided, the device comprising:

[0028] A determination module, used for determining a first environmental feature corresponding to the target intelligent agent whenever a decision cycle is reached;

[0029] An acquisition module, configured to acquire a second environmental feature corresponding to a neighboring agent of the target agent, wherein the neighboring agent is an agent preset in the multi-agent system to communicate with the target agent;

[0030] A generating module, used for generating a target fusion environment feature based on the first environment feature and the second environment feature;

[0031] The input module is used to input the target fusion environment characteristics into the policy network to obtain the target processing that the target agent needs to perform in the current decision cycle.

[0032] Optionally, the determining module is further used to:

[0033] Based on the preset corresponding relationship between the target agent and the neighboring agents, the neighboring agents of the target agent are determined.

[0034] Optionally, the generating module is used to:

[0035] Inputting the first environmental feature into an encoder to obtain a first feature to be fused;

[0036] Inputting the second environmental feature into the encoder to obtain a second feature to be fused;

[0037] The first feature to be fused and the second feature to be fused are fused to obtain a target fusion environment feature.

[0038] Optionally, the generating module is used to:

[0039] Inputting the first environmental features into a plurality of encoders respectively to obtain a plurality of third features to be fused;

[0040] Inputting the second environmental features into the multiple encoders respectively to obtain multiple fourth features to be fused;

[0041] The third feature to be fused and the fourth feature to be fused obtained by the same encoder are respectively fused to obtain a plurality of fused features to be spliced;

[0042] The multiple fusion features to be spliced ​​are spliced ​​together to obtain target fusion environment features.

[0043] Optionally, the generating module is used to:

[0044] Performing a fusion process on the first environmental feature and the second environmental feature to obtain a fusion environmental feature after a first fusion process;

[0045] Set n to 2, and obtain the fusion environment features obtained after the neighboring agents of the target agent perform the n-1th fusion process;

[0046] The fused environment features obtained after the n-1th fusion process of the neighboring intelligent bodies of the target intelligent body are fused with the fused environment features after the n-1th fusion process of the target intelligent body to obtain the fused environment features after the nth fusion process;

[0047] If n is less than N, add 1 to the value of n, and proceed to the step of obtaining the fusion environment features obtained after the n-1th fusion process of the neighboring agents of the target agent, where N is the total number of fusion processes preset, and N is greater than or equal to 2;

[0048] If n is equal to N, the fusion environment features obtained after each fusion process are spliced ​​to obtain the target fusion environment features.

[0049] Optionally, the generating module is used to:

[0050] Performing a fusion process on the first environmental feature and the second environmental feature to obtain a fusion environmental feature after a first fusion process;

[0051] Set n to 2, and determine at least one target neighboring agent corresponding to the target agent when the target agent performs the nth fusion process based on the corresponding relationship between the number of fusion processes of the target agent and the neighboring agents;

[0052] For each target neighboring agent, based on the correspondence between the number of fusion processing of the target agent and the number of fusion processing of the neighboring agent, determine the target fusion processing number corresponding to the target neighboring agent when the target agent performs the nth fusion processing, and obtain the fusion environment characteristics obtained by the target neighboring agent after performing the target fusion processing number;

[0053] The fusion environment features obtained by each target neighbor agent after the target fusion processing is performed for the number of times are fused with the fusion environment features of the target agent after the n-1th fusion processing, and the fusion environment features of the target agent after the nth fusion processing are obtained;

[0054] If n is less than N, add 1 to the value of n, and proceed to execute the correspondence between the number of fusion processing times of the target agent and the neighboring agents, and determine at least one target neighboring agent corresponding to the target agent when the target agent executes the nth fusion processing; wherein N is the total number of fusion processing preset for the target agent, and N is greater than or equal to 2;

[0055] If n is equal to N, the fusion environment features obtained after each fusion process are spliced ​​to obtain the target fusion environment features.

[0056] Optionally, the total data volume corresponding to N fusion environment features obtained after the target intelligent agent performs N fusion processes is less than or equal to a preset data volume.

[0057] In a third aspect, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the operations performed by the method for determining intelligent agent processing as described in the first aspect.

[0058] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is loaded and executed by a processor to implement the operations performed by the method for determining intelligent agent processing as described in the first aspect.

[0059] In a fifth aspect, a computer program product is provided, wherein the computer program product comprises at least one instruction, wherein the at least one instruction is loaded and executed by a processor to implement the operations performed by the method for determining intelligent agent processing as described in the first aspect.

[0060] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0061] The embodiment of the present application pre-sets the corresponding neighboring agents for the target agent. In each decision cycle of each target agent, the target agent can generate a first environmental feature based on the observed environment, and obtain a second environmental feature corresponding to the neighboring agent, and then fuse the first environmental feature and the second environmental feature to obtain a target fused environmental feature. The target fused environmental feature obtained in this way can include more environmental information. In this way, the target processing that the target agent needs to perform in the decision cycle is determined based on the target fused environmental feature with more environmental information, which can improve the accuracy of determining the processing that the agent needs to perform. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 is a flow chart of a processing method for determining an intelligent agent provided in an embodiment of the present application;

[0064] Figure 2 is a schematic diagram of a processing method for determining an intelligent agent provided in an embodiment of the present application;

[0065] Figure 3 is a flow chart of a processing method for determining an intelligent agent provided in an embodiment of the present application;

[0066] Figure 4 is a schematic diagram of a processing method for determining an intelligent agent provided in an embodiment of the present application;

[0067] Figure 5 is a flow chart of a processing method for determining an intelligent agent provided in an embodiment of the present application;

[0068] Figure 6 is a flow chart of a processing method for determining an intelligent agent provided in an embodiment of the present application;

[0069] Figure 7 is a schematic diagram of a processing method for determining an intelligent agent provided in an embodiment of the present application;

[0070] Figure 8 is a schematic diagram of a method for training a multi-agent system provided in an embodiment of the present application;

[0071] Fig. 9 is a structural diagram of a processing device for determining an intelligent agent provided in an embodiment of the present application;

[0072] Fig.10It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0074] The processing method for determining an intelligent agent provided in the embodiment of the present application belongs to the field of artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Just like the processing method for determining an intelligent agent provided in this application, it is actually the process in which the intelligent agent perceives the surrounding environment and then makes a decision based on computational reasoning.

[0075] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The following is a further introduction to the above directions:

[0076] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. For example, an artificial intelligence system can be a multi-agent system. Among them, computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0077] The key technologies of speech technology include automatic speech recognition technology (ASR), text-to-speech technology (TTS) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0078] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0079] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0080] Autonomous driving technology usually includes high-precision maps, environmental perception, behavioral decision-making, path planning, motion control and other technologies. Autonomous driving technology has broad application prospects.

[0081] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, autonomous driving, drones, robots, smart medical care, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0082] The solutions provided in the embodiments of the present application may involve artificial intelligence computer vision, machine learning and other related technologies, which are specifically explained through the following embodiments.

[0083] In the embodiments provided in the present application, the multiple agents included in the agent system may be in the form of hardware devices or software programs.

[0084] When the agent is a hardware device, the agent may include a processor, a memory, etc. The memory is used to store data related to the agent processing, for example, it may be a program code related to the agent processing, and the processor may load and process the data stored in the memory, thereby implementing the method for determining the agent processing provided by the present application, etc. In addition, the agent may also include a communication component for communicating with other agents in the multi-agent system to realize data transmission between agents. When the agent is a hardware device, the agent may be a drone, a robot, a vehicle, etc.

[0085] When the agent is a software program, the computer device running the agent may include a processor, a memory, etc. The memory is used to store data related to the agent processing, for example, it may be a program code related to the agent, and the processor may load and process the data stored in the memory, thereby implementing the method for determining the agent processing provided by the present application, etc. Multiple agents in a multi-agent system may run on the same computer device, and the computer device may start a corresponding processing process for each agent to run the corresponding agent, and may realize data transmission between agents based on communication between processes. When the agent is a software program, the agent may be an AI game character, etc.

[0086] In a multi-agent system, each agent is in the same environment, and each agent can cooperate with each other to perform the set tasks in the environment. In the process of performing tasks, the agent needs to determine the environmental characteristics corresponding to the surrounding environment according to the preset decision cycle, and then calculate the environmental characteristics through the strategy network to obtain the processing that the agent needs to perform in each decision cycle. For example, when the agent is a football robot, the playing field where the football robot is located is the environment where each agent is located, and winning the game is the task set for the football robot. The processing performed by the football robot in each decision cycle may include executing different specified actions.

[0087] The method for determining the processing of an intelligent agent provided in the embodiment of the present application can improve the efficiency of the intelligent agent in determining the processing that needs to be executed in each decision cycle, thereby improving the reaction speed of the intelligent agent. Figure 1 is a flow chart of a method for determining agent processing provided by an embodiment of the present application. Figure 1 , the embodiment comprises:

[0088] Step 101: Whenever each agent decision cycle is reached, determine the first environmental feature corresponding to the target agent.

[0089] In the embodiment of the present application, an agent in a multi-agent system is taken as an example to illustrate the method for determining agent processing provided by the embodiment of the present application. The agent may be referred to as the target agent later, and the processing of other agents in the multi-agent system is similar to that of the target agent, which will not be described in detail. In addition, it should be understood that the target agent can be a hardware device or a software program. When the target agent is a hardware device, each step in the process of the present method can be executed by the target agent. When the target agent is a software program, each step in the process of the present method can be executed by a computer device running the target agent, that is, it can be executed by a processing process opened by the computer device for the target agent.

[0090] The target agent is provided with a decision cycle. Whenever the decision cycle of the target agent is reached, the target agent can observe the surrounding environment, and then extract features of the observed environment through a feature extraction network to obtain environmental features. In the embodiment of the present application, the environmental features obtained by extracting features of the environment observed by the target agent through a feature extraction network can be referred to as first environmental features. Among them, the feature extraction network can be set by a technician according to the environment actually observed by the agent. For example, if the environment observed by the agent is an image, the feature extraction network can be an image feature extraction network, etc. The setting of the feature extraction network belongs to the prior art and will not be introduced in detail here.

[0091] When the target intelligent agent is a hardware device, this step can be executed by the target intelligent agent. For the environment observed by the target intelligent agent, the target intelligent agent can obtain the position data of the target intelligent agent through the positioning module, obtain the image data around the target intelligent agent through the shooting module, and obtain the point cloud data of the environment around the target intelligent agent through the radar module.

[0092] When the target agent is a software program, this step can be performed by a computer device running the target agent. The computer device can obtain the environment around the target agent according to the location of the target agent. For example, the target agent can be an AI game character in an electronic game, and the environment around the target agent can be the location of the AI ​​game character in the game map, other game characters within a preset distance range of the AI ​​game character, and attribute information of other game characters, etc.

[0093] Step 102: Obtain a second environmental feature corresponding to a neighboring agent of the target agent, where the neighboring agent is an agent preset in the multi-agent system to communicate with the target agent.

[0094] The technician can pre-set the communication relationship between the agents in the multi-agent system. The communication relationship between the agents can form a sparse bidirectional graph. The sparse bidirectional graph can be shown in Figure 2. In the sparse bidirectional graph, each node is an agent. The edge between the nodes is a bidirectional edge. The other agents connected to any agent through the edge are the neighboring agents of the agent.

[0095] After obtaining the first environmental feature corresponding to the target agent, the second environmental feature corresponding to the neighboring agent of the target agent can be obtained. The second environmental feature refers to the environmental feature obtained by the neighboring agent of the target agent after extracting features based on the environment it observes.

[0096] When the target agent is a hardware device, the correspondence between the target agent and the neighbor agent can be stored in the target agent. For example, the agent can be a drone, and the target drone (target agent) can store other drones (neighbor agents) with which it establishes communication. When each drone in the multi-agent system starts to perform a task, the target drone can establish communication with the neighbor drone based on the stored correspondence between the target drone and the neighbor drone. After that, in each decision cycle, the target drone can transmit the first environmental feature and the second environmental feature between the neighbor drone. For example, the target drone can send an environmental feature acquisition request to the neighbor drone, and the neighbor drone of the target drone can send the second environmental feature to the target drone after receiving the environmental feature acquisition request sent by the target drone.

[0097] When the target agent is a software object, the corresponding relationship between the target agent and the neighbor agent can be stored in the computer device running the software object. For example, the agent can be an AI game character in an electronic game, and the corresponding relationship between each AI game character can be stored in the computer device running the electronic game. When the electronic game starts, the computer device can start a corresponding processing process for each AI game character participating in the game to run each AI game character and control the AI ​​game character to perform related processing. For the target AI game character (target agent), the environment around the target AI game character can be obtained according to the decision cycle, and then the corresponding first environmental feature can be obtained. After that, the processing process corresponding to other AI game characters (neighbor agents) can be communicated between processes to obtain the second environmental features of other AI game characters, and the first environmental features can be sent to the processing processes corresponding to other AI game characters.

[0098] It can be seen that in this step, before the target agent obtains the second environmental features of the corresponding neighboring agent in each decision cycle, it is no longer necessary to calculate the first environmental features through the attention network to determine the agent that needs to communicate. It also avoids the need to establish communication with different agents in each decision cycle, reduces the amount of calculation, and can improve the efficiency of obtaining the second environmental features, thereby improving the efficiency of the agent in determining the processing that needs to be performed in each decision cycle.

[0099] It should be understood that the correspondence between the target agent and the neighboring agent can be set by the technician according to the actual application scenario and experience, and can also be set according to the maximum number of agents that a single agent can communicate with, the communication distance, etc. And the number of neighboring agents corresponding to each agent can be less than the preset number threshold, for example, the number threshold can be 3, so that in each decision cycle, each agent needs to communicate with fewer agents, and thus the amount of data that the agent needs to transmit in each decision cycle is less, so to a certain extent, it can also improve the efficiency of the agent in determining the processing that needs to be executed. And in the present application, when a new agent is added to the multi-agent system, it only needs to add the correspondence between the newly added agent and the neighboring agent in the newly added agent, so that the newly added agent can be quickly integrated into the multi-agent system.

[0100] Step 103: Generate a target fusion environment feature based on the first environment feature and the second environment feature.

[0101] After obtaining the first environmental feature corresponding to the target intelligent agent and the second environmental feature corresponding to the neighboring intelligent agent, the first environmental feature and the second environmental feature may be fused to obtain the target fused environmental feature.

[0102] In the method embodiment provided in the present application, the first environmental feature and the second environmental feature can be fused through a self-attention mechanism, and the corresponding processing is as follows: the first environmental feature is input into the encoder to obtain the first feature to be fused; the second environmental feature is input into the encoder to obtain the second feature to be fused; the first feature to be fused and the second feature to be fused are fused to obtain the target fused environmental feature.

[0103] In implementation, after obtaining the first environment feature corresponding to the target agent and the second environment feature of the target agent, the first environment feature and the second environment feature can be respectively input into the encoder in the self-attention mechanism. For the first environment feature, the encoder can generate the first feature to be fused, and for the second environment feature, the encoder can generate the second feature to be fused.

[0104] The first feature to be fused may be a query vector generated by the encoder for the first environmental feature, and the second fused feature may include a key vector and a value vector generated by the encoder for the second environmental feature. The fusion of the first feature to be fused and the second feature to be fused may be performed by the following formula:

[0105]

[0106] Among them, Q is the query vector corresponding to the first environmental feature, K i is the key vector generated for the second environment feature of the i-th neighbor agent, V i is the value vector generated for the second environmental feature of the i-th neighbor agent, n is the number of neighbor agents, softmax is the normalized exponential function, and H is the generated target fusion environmental feature.

[0107] In one possible solution, the first environmental feature and the second environmental feature generated by the encoder are fused to obtain the fused environmental feature, and then feature extraction is performed on the fused environmental feature, and then the feature after feature extraction is determined as the target fused environmental feature, wherein the feature extraction network for feature extraction here can be the same as the feature extraction network in the above step 101.

[0108] In the method embodiment provided in the present application, the first environmental feature and the second environmental feature can also be fused through a multi-headed self-attention mechanism, and the corresponding processing is as follows: the first environmental feature is input into multiple encoders respectively to obtain multiple third features to be fused; the second environmental feature is input into multiple encoders respectively to obtain multiple fourth features to be fused; the third features to be fused and the fourth features to be fused obtained by the same encoder are fused respectively to obtain multiple fused features to be spliced; the multiple fused features to be spliced ​​are spliced ​​to obtain the target fused environmental feature.

[0109] The multi-head self-attention mechanism processes the first environmental feature and the second environmental feature respectively through multiple different encoders. The third feature to be fused corresponding to the first environmental feature and the fourth feature to be fused corresponding to the second environmental feature are obtained. For the third feature to be fused and the fourth feature to be fused obtained by the same encoder, a fusion process can be performed to obtain a fusion feature to be spliced. The fusion process here is the same as the fusion process of the first fusion feature and the second fusion feature mentioned above, and will not be repeated here. The fusion feature to be spliced ​​obtained after the fusion process is equivalent to H obtained after the fusion process of the first fusion feature and the second fusion feature mentioned above. After obtaining the fusion feature to be spliced ​​corresponding to each encoder, multiple fusion features to be spliced ​​can be spliced ​​to obtain the target fusion environmental feature. It should be noted that the number of encoders can be preset by technical personnel. For example, it can be set according to the processing capability of the intelligent agent, the requirements of the multi-agent system for latency, etc.

[0110] It should be noted that in the related art, the environmental characteristics of the target agent and the neighboring agent are generally fused through the recurrent neural network. However, the stability of the recurrent neural network is too dependent on the order of input data, that is, the agent to which the first environmental feature and the second environmental feature belong, and the order of the first environmental feature and the second environmental feature input into the recurrent neural network each time need to be the same to ensure the stability of the output result. In the present application, the environmental characteristics are fused through the self-attention mechanism or the multi-head self-attention mechanism, and there is no requirement for the agent to which the first environmental feature and the second environmental feature belong, and the order of the first environmental feature and the second environmental feature. Therefore, the present application adopts the self-attention mechanism to fuse the environmental characteristics of the target agent and the neighboring agent, which can improve the stability of the fusion processing of the environmental characteristics relative to the recurrent neural network, and can make the process of multi-agent reinforcement learning simpler.

[0111] Step 104: Input the target fusion environment features into the policy network to obtain the target processing that the target agent needs to perform in the current decision cycle.

[0112] After obtaining the target fusion environment characteristics, the target fusion environment characteristics can be input into the policy network, and the policy network processes the target fusion environment characteristics to obtain the probability values ​​corresponding to all the processing that the target agent can execute. Then the processing with the largest corresponding probability value is determined as the target processing. The target processing can be executed afterwards. Among them, when the target agent is a hardware device, the corresponding processing can be the action that the target agent needs to perform in the current decision cycle, such as the moving direction, moving speed and moving distance of a robot or drone in the current decision cycle. When the agent is a software program, such as an AI game character, the corresponding processing can be the skills that need to be released in the current decision cycle.

[0113] The embodiment of the present application pre-sets the corresponding neighboring agents for the target agent. In each decision cycle of each target agent, the target agent can generate a first environmental feature based on the observed environment, and obtain a second environmental feature corresponding to the neighboring agent, and then fuse the first environmental feature and the second environmental feature to obtain a target fused environmental feature. The target fused environmental feature obtained in this way can include more environmental information. In this way, the target processing that the target agent needs to perform in the decision cycle is determined based on the target fused environmental feature with more environmental information, which can improve the accuracy of determining the processing that the agent needs to perform.

[0114] For step 103 of the present application, the present application also provides a method for obtaining target fusion environment features, see Figure 3 , the corresponding processing is as follows:

[0115] Step 301: Fusing the first environmental feature and the second environmental feature to obtain a fused environmental feature after the first fusion process.

[0116] The processing of this step may be the same as the fusion processing of the first environmental feature and the second environmental feature in step 103, and will not be repeated here.

[0117] Step 302: Set the value of n to 2.

[0118] Wherein, n is a variable in the process of this method, which can be used to indicate the number of times the fusion process is performed between agents. Since the first fusion process has been performed in step 301, the initial value of n can be set to 2. In addition, the total number of times N of fusion processes between agents can be preset by a technician. For example, it can be set according to the processing capacity of the agent, the delay requirement of the multi-agent system for the agent to perform processing in each decision cycle, etc.

[0119] Step 303: Obtain the fusion environment characteristics after the n-1th fusion processing of the neighboring intelligent body of the target intelligent body, and fuse the fusion environment characteristics after the n-1th fusion processing of the neighboring intelligent body of the target intelligent body with the fusion environment characteristics after the n-1th fusion processing of the target intelligent body to obtain the fusion environment characteristics after the nth fusion processing.

[0120] When the target agent performs the second and subsequent fusion processing, it is actually to perform the fusion processing again on the fusion environment features obtained by the target agent and the neighboring agents in the previous fusion processing. In this way, as the number of fusion processing increases, the fusion environment features obtained by the target agent include more and more environmental feature information.

[0121] like Figure 4As shown, the neighbor value agents of the target agent 0 include agent 1 and agent 2, the neighbor agents of agent 1 include agent 11 and agent 12, and the neighbor agents of agent 2 include agent 21 and agent 22. In this way, the fused environment features obtained after the target agent 0 performs the first fusion processing include the environment information corresponding to the target agent 0, agent 1 and agent 2. Similarly, agents 1, 2, etc. will also perform the first fusion processing. The fused environment features obtained after the first fusion processing of agent 1 include the environment information corresponding to agent 1, agent 11 and agent 12, and the fused environment features obtained after the first fusion processing of agent 2 include the environment information corresponding to agent 2, agent 21 and agent 22. In this way, since the target agent 0 performs the second fusion process, it performs the fusion process again on the fusion environment features obtained by its neighboring agents, that is, agents 1 and 2 after the first fusion process. Therefore, after the second fusion process, the fusion environment features corresponding to the target agent 0 may include the corresponding environment information of agents 1, 11, 12, 2, 21, and 22. It can be seen that as the number of fusion processes increases, the environmental feature information included in the fusion environment features corresponding to each agent increases. In this way, for a single agent, it can obtain the environmental feature information of more agents through multiple communications when there are few communication channels (that is, few agents communicating with it).

[0122] Step 304: If n is less than N, add 1 to the value of n and proceed to the step of obtaining the fusion environment features obtained after the n-1th fusion processing of the neighboring agents of the target agent.

[0123] If n is less than N, it means that all fusion processing of the target agent has not been completed. Therefore, the value of n can be increased by 1 to perform the next fusion processing, that is, go to step 303.

[0124] Step 305: If n is equal to N, the fusion environment features obtained after each fusion process are spliced ​​to obtain the target fusion environment features.

[0125] If n is equal to N, it means that all fusion processing of the target intelligent agent has been completed, and the fusion environment features after each fusion processing can be spliced ​​to obtain the target fusion environment features. Among them, for different fusion environment features, the splicing coefficient can be set according to the number of fusion processing corresponding to the fusion environment feature, and then the fusion environment features after each fusion processing can be spliced ​​according to the number of fusion processing and the splicing coefficient corresponding to each fusion environment feature.

[0126] like Figure 5 As shown, Figure 5It is a schematic diagram of the process of obtaining the target fusion environment features of this method. After observing the environment, any agent can input the observed environment into the feature extraction network to obtain the corresponding environment features, which can exist in the mathematical form of feature vectors. Then, the neighbor feature vector of the agent's neighbor value agent can be obtained, and the feature of the agent is adjacent to the environment features of the neighbor agent to obtain the attention fusion vector, and then the attention fusion vector is extracted through the feature extraction network to obtain a fusion feature vector after a fusion process. After that, the neighbor feature vector of the neighbor agent corresponding to the agent can be obtained after a fusion process, and feature vector fusion, feature extraction and other processes can be performed repeatedly to obtain the final fusion process vector, and the final fusion process vector is input into the strategy network to obtain the process that the agent needs to perform in the current decision cycle.

[0127] The target fusion environment features corresponding to the target agent obtained in this way can include the environment information corresponding to the neighboring agents of the target agent, and can also include the environment information corresponding to the neighboring agents of the non-target agent. In this way, the target fusion environment features are input into the policy network, and the policy network can determine the processing that the target agent needs to perform based on more environmental information.

[0128] For step 103 of the present application, the present application also provides another method for obtaining the target fusion environment characteristics, see Figure 6 , the corresponding processing is as follows:

[0129] Step 601: Fusing the first environmental feature and the second environmental feature to obtain a fused environmental feature after the first fusion process.

[0130] The processing of this step may be the same as the fusion processing of the first environmental feature and the second environmental feature in step 103, and will not be repeated here.

[0131] Step 602: Set the value of n to 2.

[0132] Among them, n is a variable in the process of this method, which can be used to represent the number of times the fusion process is performed between the agents. Since the first fusion process has been performed in step 601, the initial value of n can be set to 2. In addition, the total number of times each agent performs fusion processing is N, and the value of N corresponding to different agents can be different, which can be preset by the technician. For example, according to the processing capacity of each agent, the multi-agent system can set the delay requirements for the agent to execute the processing in each decision cycle.

[0133] Step 603: Based on the correspondence between the number of fusion processing times of the target agent and the neighboring agents, determine at least one target neighboring agent corresponding to the target agent when the target agent performs the nth fusion processing.

[0134] In the process of this method, the target agent may have different neighbor agents corresponding to each fusion process in the same decision cycle due to different processing capabilities of the agents. For example, when the target agent performs the first fusion process, the corresponding neighbor agents are agent A, agent B, and agent C. It may be that the processing capability of agent A is low, resulting in a slow fusion process. Therefore, when the target agent performs the second fusion process, the corresponding neighbor agents may be agent B and agent C. Therefore, the technician can pre-set the corresponding relationship between the target agent and the neighbor agents at different fusion processing times. Before each fusion process, the target agent can determine the neighbor agent corresponding to the target agent in the corresponding relationship according to the number of fusion processes. And if there are different neighbor agents corresponding to each fusion process, the fusion environment features obtained after multiple fusion processes can include more environment features corresponding to agents.

[0135] It should be noted that even if the target agent has different neighbor agents corresponding to it each time the fusion process is performed, in a decision cycle, the total number of neighbor agents corresponding to the target agent in different fusion processes can be controlled within the set number threshold, that is, the total number of neighbor agents corresponding to the target agent is less than or equal to the set number threshold. In this way, the number of times the target agent establishes communication with the neighbor agents can be reduced, thereby reducing the delay caused by establishing communication with the neighbor agents and improving the efficiency of the fusion process.

[0136] Step 604: For each target neighbor intelligent agent, based on the correspondence between the number of fusion processing of the target intelligent agent and the number of fusion processing of the neighbor intelligent agent, determine the target fusion processing number corresponding to the target neighbor intelligent agent when the target intelligent agent performs the nth fusion processing, and obtain the fusion environment characteristics obtained by the target neighbor intelligent agent after performing the target fusion processing number.

[0137] Different agents may have different processing capabilities, and different agents may need to fuse different amounts of environmental feature data when performing fusion processing, which results in different fusion processing speeds for different agents. In this way, different agents perform fusion processing at different times in the same period of time.

[0138] Therefore, the technician can pre-set the corresponding relationship between the number of fusion processing of the target agent and the number of fusion processing of the neighboring agent. Before the target agent performs each fusion processing, the fusion environment features of the neighboring agent after the fusion processing of the target agent are determined according to the number of fusion processing of the target agent and the corresponding relationship. Then, a fusion environment feature acquisition request is sent to the corresponding neighboring agent to obtain the corresponding fusion environment features.

[0139] Step 605: fuse the fusion environment features obtained by each target neighbor agent after the target fusion processing is performed for the number of times with the fusion environment features of the target agent after the n-1th fusion processing to obtain the fusion environment features of the target agent after the nth fusion processing.

[0140] After obtaining the fusion environment characteristics of the neighboring intelligent agents at the corresponding number of fusion processing times, the obtained fusion environment characteristics of the neighboring intelligent agents can be fused with the fusion environment characteristics of the target intelligent agent after the n-1th fusion processing to obtain the fusion environment characteristics of the target intelligent agent after the nth fusion processing.

[0141] like Figure 7 As shown, in the first fusion process of the target agent 0, the corresponding neighbor value agents include agent 1 and agent 2, and in the second fusion process, the corresponding neighbor value agents include agent 3 and agent 4. Among them, the neighbor agents of agent 3 include agent 31 and agent 32, and agent 4 includes agent 41 and agent 42. In the first fusion process, the target agent 0 obtains the environmental features corresponding to agent 1 and agent 2 respectively, and the obtained fusion environment vector includes the environmental information corresponding to the target agent 0, agent 1, and agent 2. In the second fusion process, the target agent 0 obtains the fusion environment features of agent 3 and agent 4 after the first fusion process respectively, so that the obtained fusion environment vector includes the environmental information corresponding to the target agent 0, agent 1, agent 2, agent 3, agent 4, agent 31, agent 32, agent 41, and agent 42. It can be seen that as the number of fusion processes increases, the environmental feature information included in the fusion environment features corresponding to each agent is increasing. In this way, for a single intelligent agent, when there are few communication channels (i.e., few intelligent agents to communicate with), it can obtain environmental feature information of more intelligent agents through multiple communications with neighboring intelligent agents.

[0142] Step 606: If n is less than N, add 1 to the value of n, and execute the corresponding relationship between the number of fusion processing times of the target intelligent agent and the neighboring intelligent agents, and determine at least one target neighboring intelligent agent corresponding to the target intelligent agent when the target intelligent agent performs the nth fusion processing.

[0143] If n is less than N, it means that all fusion processing of the target agent has not been completed. Therefore, the value of n can be increased by 1 to perform the next fusion processing, that is, go to step 703.

[0144] Step 607: If n is equal to N, the fusion environment features obtained after each fusion process are spliced ​​to obtain the target fusion environment features.

[0145] If n is equal to N, it means that all fusion processing of the target intelligent agent has been completed, and the fusion environment features after each fusion processing can be spliced ​​to obtain the target fusion environment features. Among them, for different fusion environment features, the splicing coefficient can be set according to the number of fusion processing corresponding to the fusion environment feature, and then the fusion environment features after each fusion processing can be spliced ​​according to the number of fusion processing and the splicing coefficient corresponding to each fusion environment feature.

[0146] The target fusion environment features corresponding to the target agent obtained in this way can include the environment information corresponding to the neighboring agents of the target agent, and can also include the environment information corresponding to the neighboring agents of the non-target agent. In this way, the target fusion environment features are input into the policy network, and the policy network can determine the processing that the target agent needs to perform based on more environmental information.

[0147] In one possible scenario, Figure 3 or Figure 6 In the embodiment, the total amount of data corresponding to the fusion environment features obtained by the target intelligent agent after multiple fusion processes are performed is less than a preset data amount threshold.

[0148] In implementation, it is constrained by the communication capability, processing capability and latency requirements of the agent system. Each time the agent communicates, it is necessary to reduce the amount of data transmitted each time as much as possible, thereby reducing the latency when transmitting data and improving the efficiency of determining the processing that the target agent needs to perform. Among them, the amount of data transmitted between agents is directly related to the dimension of the fusion environment features generated after the agent performs fusion processing. The higher the dimension of the fusion environment features, the larger the amount of data corresponding to the fusion environment features. Therefore, by setting the dimension of the output data of the feature fusion model that performs different fusion processing times, setting the amount of data of the output fusion environment features, and controlling the total amount of data of the fusion environment features obtained after multiple fusion processing to be less than the preset data volume threshold, the time occupied by data transmission in the decision cycle is controlled to ensure the efficiency of the processing that the target agent needs to perform.

[0149] The technicians can set the dimension of the fusion environment features after each fusion process according to the actual business needs. For example, in one possible situation, when the agent performs the fusion process for the first few times, the environmental features or fusion environment features corresponding to the corresponding neighboring agents have a greater impact on the target agent. Therefore, the fusion environment features generated by each agent after the first few fusion processes can have a higher dimension, and the dimension of the fusion environment features of each agent after the subsequent fusion processes can be reduced.

[0150] The embodiment of the present application pre-sets the corresponding neighboring agents for the target agent. In each decision cycle of each target agent, the target agent can generate a first environmental feature based on the observed environment, and obtain a second environmental feature corresponding to the neighboring agent, and then fuse the first environmental feature and the second environmental feature to obtain a target fused environmental feature. The target fused environmental feature obtained in this way can include more environmental information. In this way, the target processing that the target agent needs to perform in the decision cycle is determined based on the target fused environmental feature with more environmental information, which can improve the accuracy of determining the processing that the agent needs to perform.

[0151] Figure 8 This is a flow chart of a method for training a multi-agent system provided in an embodiment of the present application, see Figure 8 , the training method includes:

[0152] Step 801: Whenever each agent reaches a decision cycle, determine the third environmental feature corresponding to the target agent, and obtain the fourth environmental feature corresponding to the neighboring agents of the target agent.

[0153] The processing of step 801 can refer to the processing of step 101 and step 102, and the corresponding feature extraction network in this step can be an untrained feature extraction network. The neighboring agents of the target agent can be pre-set by a technician.

[0154] Step 802: Generate fusion environment features corresponding to the target intelligent agent based on the third environment features and the fourth environment features.

[0155] The processing of step 802 can refer to the processing of step 103, that is, the third environment feature and the fourth environment feature corresponding to the target intelligent agent can be fused by one or more encoders, or by Figure 3 The method shown performs fusion processing on the third environmental feature and the fourth environmental feature. In this step, the encoder can be an untrained encoder.

[0156] Step 803: Input the fusion environment features corresponding to the target intelligent agent into the policy network to obtain the probability values ​​corresponding to the target intelligent agent executing each process.

[0157] After obtaining the fusion environment features corresponding to the target intelligent agent, the fusion environment features corresponding to the target intelligent agent can be respectively input into the policy network corresponding to the target intelligent agent, and the policy network corresponding to the target intelligent agent outputs the probability values ​​of various processing that can be executed by the corresponding target intelligent agent.

[0158] Step 804: The target intelligent agent randomly executes any of the processes, and after executing any of the processes, sends the processing information corresponding to any of the processes and the probability values ​​corresponding to each of the processes in step 803 to the central node.

[0159] The processing information corresponding to any process may be a processing identifier corresponding to the process.

[0160] Step 805: The central node receives the processing information corresponding to the first processing sent by each intelligent agent, and the probability value corresponding to each intelligent agent executing each processing, and trains each intelligent agent based on the processing information corresponding to the first processing sent by each intelligent agent, and the probability value corresponding to each intelligent agent executing each processing.

[0161] In this application, the weights of the feature extraction network, encoder and policy network corresponding to each agent are trained by the actor-judge model. A central node can be introduced in the training process. Among them, when each agent is a hardware device, the central node can be a separate hardware device that can communicate with all agents. It is used to perform the processing of this step, or one of the multiple agents of the central node. When each agent is a software program, the central node can be a processing process in the computer device running each software program, which is used to perform all the processing of this step.

[0162] The central node can receive the processing information corresponding to the executed processing sent by each agent, the probability value corresponding to the target agent executing each processing, and can obtain the environmental data observed by all agents, the benefit feedback given by the environment after all agents execute the corresponding processing, and set the following Q function, by training the feature extraction network, encoder and strategy network of each agent and the judge network of the central node respectively.

[0163] Q π (s,a1,a2,...,a N )=E s′ (r(S,a1,a2,...,a N )+γE a′～π (Q π (S′, a′1, a′2,..., a′ N ))

[0164] In the above Q function, a1-a Nis the processing performed by N agents in a decision cycle, N is the total number of agents, and π is a probability matrix, which consists of the probability vectors corresponding to the different processing performed by each agent. s represents the set of environments observed by each agent, for example, all the information observed by all AI game characters, or images captured by drones through camera devices. Q π (s,a1,a2,...,a N ) represents the total benefit generated by the system after each agent performs each process in the observed environment. r(s, a1, a2, ..., a N ) means that each agent executes a1-a in sequence N The environmental benefits generated after processing can be pre-set by the technicians, or generated by the environment itself. For example, the benefits can be the total gold coins or total experience points generated in the game after each AI game character performs each action, or the task completion degree set by the technicians after each action performed by the drone, or the technicians can pre-set the corresponding relationship between different actions and environmental benefits, and determine the execution of a1-a by each agent in sequence according to the corresponding relationship. N The environmental benefits generated after processing. γ is the preset decreasing weight, s′ is the agent’s N The treated environment, a′1-a′ N E is the actions performed by N agents according to their corresponding probabilities. a′～π (Q π (s′,a′1,a′2,...,a′ N )) represents the predicted benefit obtained by summing up the probability value corresponding to each processing and the preset benefit corresponding to each processing after each intelligent agent executes each processing separately.

[0165] In order to better represent the Q function, the central node uses a neural network with a parameter θ to fit the Q function, with a = (a1, a2, ..., a N )Simplify the Q function and get the following expression:

[0166]

[0167] in, r(s, a) is the agent executing a1-a in sequence N Environmental benefits after treatment, To predict the return, Q π (s, a|θ) is the output of the judge network with the central node. Each agent performs a1-a N Environmental benefits generated after treatment. It can be updated periodically, and can randomly sample the probability of each agent executing each action, and then perform weighted calculation on the preset benefits and corresponding probability values ​​corresponding to the randomly sampled actions to obtain the predicted benefits. After obtaining the simplified Q function, the training formula can be constructed based on the Q function and the policy gradient algorithm as follows:

[0168]

[0169] Where φ represents the set of all parameters in the policy network, feature extraction network, and encoder in the self-attention mechanism. The above training formula is the gradient calculation formula for φ, where It is used to represent the policy network, a is the sampling action, and o is the input of the policy network. i represents the observation of the ith agent, o j It represents the observations of all agents corresponding to the feature information that the ith agent can obtain after feature fusion processing. The above training formula can be optimized by the gradient descent algorithm to achieve the training of the agent.

[0170] In order to stabilize the training process, a counterfactual baseline is usually added to reduce the noise of other agents on the gradient of the i-th agent. Using a counterfactual baseline to replace the Q function in the gradient formula can effectively reduce the noise error caused by other agents, so that each parameter can be trained more efficiently. The formula is as follows:

[0171]

[0172] in, represents the weighted benefit calculated by the agent according to the probability corresponding to each treatment, a i ′ represents the processing performed by the i-th agent enumerated.

[0173] After training the strategy networks, feature extraction networks, encoders, etc. corresponding to the multiple agents for multiple times through the above training method, a trained agent can be obtained. In this way, for any trained agent, the method for determining agent processing provided by the present application can be used to determine the processing corresponding to each decision cycle, which can improve the efficiency of the agent in determining the processing to be executed in the decision cycle.

[0174] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0175] Fig. 9 is a device for determining agent processing provided by an embodiment of the present application. When the agent is a hardware device, the device may be the agent in the above embodiment. When the agent is a software program, the device may be a computer device running the agent. Fig. 9 , the device comprises:

[0176] A determination module 910, configured to determine a first environmental feature corresponding to the target agent whenever a decision cycle is reached;

[0177] An acquisition module 920 is used to acquire a second environmental feature corresponding to a neighboring agent of the target agent, where the neighboring agent is an agent preset in the multi-agent system to communicate with the target agent;

[0178] A generating module 930, configured to generate a target fusion environment feature based on the first environment feature and the second environment feature;

[0179] The input module 940 is used to input the target fusion environment characteristics into the policy network to obtain the target processing that the target agent needs to perform in the current decision cycle.

[0180] Optionally, the determining module 910 is further configured to:

[0181] Based on the preset corresponding relationship between the target agent and the neighboring agents, the neighboring agents of the target agent are determined.

[0182] Optionally, the generating module 930 is used to:

[0183] Inputting the first environmental feature into an encoder to obtain a first feature to be fused;

[0184] Inputting the second environmental feature into the encoder to obtain a second feature to be fused;

[0185] The first feature to be fused and the second feature to be fused are fused to obtain a target fusion environment feature.

[0186] Optionally, the generating module 930 is used to:

[0187] Inputting the first environmental features into a plurality of encoders respectively to obtain a plurality of third features to be fused;

[0188] Inputting the second environmental features into the multiple encoders respectively to obtain multiple fourth features to be fused;

[0189] The third feature to be fused and the fourth feature to be fused obtained by the same encoder are respectively fused to obtain a plurality of fused features to be spliced;

[0190] The multiple fusion features to be spliced ​​are spliced ​​together to obtain target fusion environment features.

[0191] Optionally, the generating module 930 is used to:

[0192] Performing a fusion process on the first environmental feature and the second environmental feature to obtain a fusion environmental feature after a first fusion process;

[0193] Set n to 2, and obtain the fusion environment features obtained after the neighboring agents of the target agent perform the n-1th fusion process;

[0194] The fused environment features obtained after the n-1th fusion process of the neighboring intelligent bodies of the target intelligent body are fused with the fused environment features after the n-1th fusion process of the target intelligent body to obtain the fused environment features after the nth fusion process;

[0195] If n is less than N, add 1 to the value of n, and proceed to the step of obtaining the fusion environment features obtained after the n-1th fusion process of the neighboring agents of the target agent, where N is the total number of fusion processes preset, and N is greater than or equal to 2;

[0196] If n is equal to N, the fusion environment features obtained after each fusion process are spliced ​​to obtain the target fusion environment features.

[0197] Optionally, the generating module 930 is used to:

[0198] Performing a fusion process on the first environmental feature and the second environmental feature to obtain a fusion environmental feature after a first fusion process;

[0199] Set n to 2, and determine at least one target neighboring agent corresponding to the target agent when the target agent performs the nth fusion process based on the corresponding relationship between the number of fusion processes of the target agent and the neighboring agents;

[0200] For each target neighboring agent, based on the correspondence between the number of fusion processing of the target agent and the number of fusion processing of the neighboring agent, determine the target fusion processing number corresponding to the target neighboring agent when the target agent performs the nth fusion processing, and obtain the fusion environment characteristics obtained by the target neighboring agent after performing the target fusion processing number;

[0201] The fusion environment features obtained by each target neighbor agent after the target fusion processing is performed for the number of times are fused with the fusion environment features of the target agent after the n-1th fusion processing, and the fusion environment features of the target agent after the nth fusion processing are obtained;

[0202] If n is less than N, add 1 to the value of n, and proceed to execute the correspondence between the number of fusion processing times of the target agent and the neighboring agents, and determine at least one target neighboring agent corresponding to the target agent when the target agent executes the nth fusion processing; wherein N is the total number of fusion processing preset for the target agent, and N is greater than or equal to 2;

[0203] If n is equal to N, the fusion environment features obtained after each fusion process are spliced ​​to obtain the target fusion environment features.

[0204] Optionally, the total data volume corresponding to N fusion environment features obtained after the target intelligent agent performs N fusion processes is less than or equal to a preset data volume.

[0205] The embodiment of the present application pre-sets the corresponding neighboring agents for the target agent. In each decision cycle of each target agent, the target agent can generate a first environmental feature based on the observed environment, and obtain a second environmental feature corresponding to the neighboring agent, and then fuse the first environmental feature and the second environmental feature to obtain a target fused environmental feature. The target fused environmental feature obtained in this way can include more environmental information. In this way, the target processing that the target agent needs to perform in the decision cycle is determined based on the target fused environmental feature with more environmental information, which can improve the accuracy of determining the processing that the agent needs to perform.

[0206] It should be noted that: the device for determining the intelligent agent processing provided in the above embodiment only uses the division of the above functional modules as an example when determining the intelligent agent processing. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device (intelligent agent or computer device running the intelligent agent) is divided into different functional modules to complete all or part of the functions described above. In addition, the device for determining the intelligent agent processing provided in the above embodiment and the method embodiment for determining the intelligent agent processing belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0207] Fig.10It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. When the agent is a hardware device, the computer device 1000 can be the agent in the above embodiment. When the agent is a software program, the computer device 1000 can be used to run the agent. The computer device 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1001 and one or more memories 1002. Among them, at least one instruction is stored in the memory 1002, and the at least one instruction is loaded and executed by the processor 1001 to implement the methods provided by the above-mentioned various method embodiments. Of course, the computer device 1000 may also have components such as a wired or wireless network interface, various input and output interfaces, etc., for input and output. The computer device 1000 may also include other components for implementing device functions, which will not be repeated here.

[0208] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the method of determining agent processing in the above embodiment. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0209] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned computer-readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0210] In an exemplary embodiment, a computer program product is also provided, which includes at least one instruction, and the at least one instruction can be loaded and executed by a processor to implement the method for determining intelligent agent processing provided in the present application.

[0211] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the environmental features involved in this application are all obtained with full authorization.

[0212] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining agent processing, characterized in that The method comprises: Whenever a decision cycle is reached, determining a first environmental feature corresponding to the target agent; Acquire a second environmental feature corresponding to a neighboring agent of the target agent, wherein the neighboring agent is an agent preset in the multi-agent system to communicate with the target agent; Based on the first environmental feature and the second environmental feature, a target fusion environmental feature is generated through multiple fusion processes. In each fusion process, based on the number of fusion processes of the target agent and the corresponding relationship, at least one target neighbor agent corresponding to the target agent when the target agent performs this fusion process and the number of target fusion processes corresponding to the target neighbor agent are determined, and the fusion environmental feature obtained by the target neighbor agent after performing the target fusion process times is obtained. The fusion environmental feature obtained by the target neighbor agent after performing the target fusion process times is fused with the fusion environmental feature after the last fusion process of the target agent, and the corresponding relationship is set based on the speed of fusion processing performed by different agents; The target fusion environment features are input into the policy network to obtain the target processing that the target agent needs to perform in the current decision cycle.

2. The method according to claim 1, characterized in that Before obtaining the second environment feature corresponding to the neighboring agent of the target agent, the method further includes: Based on the preset corresponding relationship between the target agent and the neighboring agents, the neighboring agents of the target agent are determined.

3. The method according to claim 1, characterized in that: The generating a target fusion environment feature based on the first environment feature and the second environment feature includes: Inputting the first environmental feature into an encoder to obtain a first feature to be fused; Inputting the second environmental feature into the encoder to obtain a second feature to be fused; The first feature to be fused and the second feature to be fused are fused to obtain a target fusion environment feature.

4. The method according to claim 1, characterized in that: The generating a target fusion environment feature based on the first environment feature and the second environment feature includes: Inputting the first environmental features into a plurality of encoders respectively to obtain a plurality of third features to be fused; Inputting the second environmental features into the multiple encoders respectively to obtain multiple fourth features to be fused; The third feature to be fused and the fourth feature to be fused obtained by the same encoder are respectively fused to obtain a plurality of fused features to be spliced; The multiple fusion features to be spliced ​​are spliced ​​together to obtain target fusion environment features.

5. The method according to claim 1, characterized in that The generating a target fusion environment feature based on the first environment feature and the second environment feature includes: Performing a fusion process on the first environmental feature and the second environmental feature to obtain a fusion environmental feature after a first fusion process; Set n to 2, and obtain the fusion environment features obtained after the neighboring agents of the target agent perform the n-1th fusion process; The fused environment features obtained after the n-1th fusion processing of the neighboring intelligent bodies of the target intelligent body are fused with the fused environment features after the n-1th fusion processing of the target intelligent body to obtain the fused environment features after the nth fusion processing; If n is less than N, add 1 to the value of n, and proceed to the step of obtaining the fusion environment features obtained after the n-1th fusion process of the neighboring agents of the target agent, where N is the total number of fusion processes preset, and N is greater than or equal to 2; If n is equal to N, the fusion environment features obtained after each fusion process are spliced ​​to obtain the target fusion environment features.

6. The method according to claim 1, characterized in that The generating a target fusion environment feature based on the first environment feature and the second environment feature includes: Performing a fusion process on the first environmental feature and the second environmental feature to obtain a fusion environmental feature after a first fusion process; Set n to 2, and determine at least one target neighboring agent corresponding to the target agent when the target agent performs the nth fusion process based on the corresponding relationship between the number of fusion processes of the target agent and the neighboring agents; For each target neighboring agent, based on the correspondence between the number of fusion processing of the target agent and the number of fusion processing of the neighboring agent, determine the target fusion processing number corresponding to the target neighboring agent when the target agent performs the nth fusion processing, and obtain the fusion environment characteristics obtained by the target neighboring agent after performing the target fusion processing number; The fusion environment features obtained by each target neighbor agent after the target fusion processing is performed for the number of times are fused with the fusion environment features of the target agent after the n-1th fusion processing, and the fusion environment features of the target agent after the nth fusion processing are obtained; If n is less than N, add 1 to the value of n, and proceed to execute the correspondence between the number of fusion processing times of the target agent and the neighboring agents, and determine at least one target neighboring agent corresponding to the target agent when the target agent executes the nth fusion processing; wherein N is the total number of fusion processing preset for the target agent, and N is greater than or equal to 2; If n is equal to N, the fusion environment features obtained after each fusion process are spliced ​​to obtain the target fusion environment features.

7. The method according to any one of claims 5-6, characterized in that: The total data volume corresponding to N fusion environment features obtained after the target intelligent agent performs N fusion processes is less than or equal to the preset data volume.

8. A device for determining agent processing, characterized in that The device comprises: A determination module, used for determining a first environmental feature corresponding to the target intelligent agent whenever a decision cycle is reached; An acquisition module, configured to acquire a second environmental feature corresponding to a neighboring agent of the target agent, wherein the neighboring agent is an agent preset in the multi-agent system to communicate with the target agent; A generation module, for generating target fusion environment features through multiple fusion processes based on the first environment features and the second environment features, and in each fusion process, determining at least one target neighboring agent corresponding to the target agent when the target agent performs this fusion process and the target fusion process times corresponding to the target neighboring agent based on the fusion process times and the corresponding relationship, obtaining the fusion environment features obtained by the target neighboring agent after performing the target fusion process times, and fusing the fusion environment features obtained by the target neighboring agent after performing the target fusion process times with the fusion environment features after the last fusion process of the target agent, wherein the corresponding relationship is set based on the speed of fusion process performed by different agents; The input module is used to input the target fusion environment characteristics into the policy network to obtain the target processing that the target agent needs to perform in the current decision cycle.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the operations performed by the method for determining intelligent agent processing as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the operations performed by the method for determining intelligent agent processing as described in any one of claims 1 to 7.

Images