Method and apparatus for belief graph construction and agent training for target environment

Abstract

A belief map construction method of a target environment comprises: for each agent in a set of agents, obtaining state information of a set of movable units, including a current state estimate and uncertainty of each movable unit in the agent's perspective; for each grid of the target environment, determining a first influence degree representing an influence of each movable unit in the agent's perspective on the grid according to the state information, to obtain a first plurality of first influence degrees; obtaining feature information of the set of agents, including a current position and a detection parameter of each agent; for each grid, determining a second influence degree representing an influence of each agent on the grid according to the feature information, to obtain a second plurality of second influence degrees; determining a belief value indicating a possibility of existence of a movable unit in the grid according to the first plurality of first influence degrees and the second plurality of second influence degrees of each grid, to obtain a set of belief values of the target environment; and constructing a belief map of the target environment according to the set of belief values.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to a method and device for belief map construction and agent training of a target environment, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] With the development of technology, modern warfare is gradually moving towards electronic and information-based. How to obtain accurate and complete combat environment information becomes particularly important. In this regard, environment modeling has become one of the key research works. Efficient representation of the distribution of movable units is an important part of environment modeling. Traditional environment modeling methods are usually based on vectorization representation. From the perspective of the observer, the vectorization representation method loses the spatial information of the environment.

[0003] The methods described in this section can not necessarily be the methods previously conceived or employed. Unless otherwise indicated, nothing in this section should be assumed to be prior art merely because of its inclusion in this section. Similarly, issues mentioned in this section should not be assumed to have been admitted to be prior art in any jurisdiction merely because of their inclusion in this section. SUMMARY

[0004] The present disclosure provides a method and device for belief map construction and agent training of a target environment, an electronic device, and a computer readable storage medium.

[0005] According to an aspect of the present disclosure, a method for belief map construction of a target environment is provided, comprising: for each agent in a set of agents in the target environment, obtaining state information of a set of movable units in the target environment, the state information including a current state estimate and an associated uncertainty of each movable unit in the set of movable units from the perspective of the agent; for each grid of the target environment, determining a first influence degree representing an influence of each movable unit on the grid from the perspective of the agent according to the state information of the set of movable units, to obtain a first plurality of first influence degrees; obtaining feature information of the set of agents, the feature information including a current position and a detection parameter of each agent in the set of agents; for each grid of the target environment, determining a second influence degree representing an influence of each agent on the grid according to the feature information of the set of agents, to obtain a second plurality of second influence degrees; determining a belief value indicating a possibility of the existence of a movable unit in each grid of the target environment according to the first plurality of first influence degrees and the second plurality of second influence degrees of each grid of the target environment, to obtain a set of belief values of the target environment; and constructing a belief map of the target environment according to the set of belief values, the belief map including color blocks respectively associated with each grid of the target environment, wherein there is a correspondence between the belief value of the grid associated with the color block and the color of the color block.

[0006] According to another aspect of the present disclosure, there is provided an agent training method, comprising: in a sample collection phase, performing the following operations: for a current time step and a next time step, constructing a respective belief graph of a target environment according to the method of the present disclosure; inputting the respective belief graph into a multi-layer convolutional neural network to obtain a respective plurality of sub-sampling feature vectors output by the multi-layer convolutional neural network; determining a respective fused feature vector as the current state and the next state of the target environment according to the respective plurality of sub-sampling feature vectors using an attention mechanism; causing a selected policy of a target reinforcement learning model to interact with the target environment at the current time step to obtain an action to be taken by a set of agents in the current state, the action causing each agent in the set of agents to move from a respective current position at the current time step to a respective position at the next time step; storing the current state, the action to be taken by the set of agents in the current state, a reward obtained by the set of agents taking the action in the current state, and the next state as a four-tuple training sample into an experience pool, wherein the number of four-tuple training samples required for one episode is denoted as a first sample number; and in response to the number of newly stored four-tuple training samples in the experience pool reaching a preset multiple of the first sample number, discontinuing the operation of the current sample collection phase and entering a model training phase; or in response to the number of newly stored four-tuple training samples in the experience pool not reaching the preset multiple of the first sample number, continuing the above operation, in the model training phase, performing the following operations: updating the policy of the reinforcement learning model using a batch of four-tuple training samples selected from the experience pool; taking the updated policy of the reinforcement learning model as the policy trained by the reinforcement learning model in the current model training phase; and comparing the policy trained in the current model training phase with one or more policies trained in previous model training phases to determine the selected policy of the target reinforcement learning model in the next sample collection phase.

[0007] According to another aspect of the present disclosure, there is provided a belief map construction apparatus for a target environment, comprising: a first obtaining module configured to, for each agent in a set of agents in the target environment, obtain state information of a set of movable units in the target environment, the state information comprising a current state estimate and an associated uncertainty for each movable unit in the set of movable units from the perspective of the agent; a first determining module configured to, for each grid of the target environment, determine, based on the state information of the set of movable units, a first influence degree representing an influence of each movable unit in the set of movable units on the grid from the perspective of the agent, to obtain a first plurality of first influence degrees; a second obtaining module configured to obtain feature information of the set of agents, the feature information comprising a current position and a detection parameter of each agent in the set of agents; a second determining module configured to, for each grid of the target environment, determine, based on the feature information of the set of agents, a second influence degree representing an influence of each agent on the grid, to obtain a second plurality of second influence degrees; a third determining module configured to determine, based on the first plurality of first influence degrees and the second plurality of second influence degrees for each grid of the target environment, a belief value indicating a likelihood of presence of a movable unit within the grid, to obtain a set of belief values for the target environment; and a construction module configured to construct a belief map for the target environment based on the set of belief values, the belief map comprising color blocks respectively associated with the grids of the target environment, wherein there is a correspondence between the belief value of the grid associated with a color block and the color of the color block.

[0008] According to another aspect of the present disclosure, there is provided an agent training apparatus, comprising: a sample collection module comprising: a first unit configured to construct, for a current time step and a next time step, a respective belief graph of a target environment according to the method of the present disclosure; a second unit configured to input the respective belief graph into a multi-layer convolutional neural network to obtain a respective plurality of sub-sampling feature vectors output by the multi-layer convolutional neural network; a third unit configured to determine, using an attention mechanism, a respective fused feature vector as a current state and a next state of the target environment according to the respective plurality of sub-sampling feature vectors; a fourth unit configured to cause a selected policy of a target reinforcement learning model to interact with the target environment at the current time step to obtain an action to be taken by a set of agents in the current state, the action causing each agent in the set of agents to move from a respective current position at the current time step to a respective position at the next time step; a fifth unit configured to store, as a four-tuple training sample, the current state, the action to be taken by the set of agents in the current state, a reward obtained by the set of agents taking the action in the current state, and the next state into an experience pool, wherein a number of four-tuple training samples required for one episode is denoted as a first sample number; and a sixth unit configured to, in response to a number of newly stored four-tuple training samples in the experience pool reaching a preset multiple of the first sample number, interrupt an operation of the current sample collection stage and enter a model training stage; and a seventh unit configured to, in response to the number of newly stored four-tuple training samples in the experience pool not reaching the preset multiple of the first sample number, continue the operation, a model training module comprising: an eighth unit configured to update a policy of the reinforcement learning model using a batch of four-tuple training samples selected from the experience pool; a ninth unit configured to use the updated policy of the reinforcement learning model as a policy of the reinforcement learning model trained in the current model training stage; and a tenth unit configured to compare the policy trained in the current model training stage with one or more policies trained in previous model training stages to determine a selected policy of the target reinforcement learning model in a next sample collection stage.

[0009] According to yet another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program which, when executed by the at least one processor, implements the above method.

[0010] According to yet another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the above method.

[0011] According to yet another aspect of the present disclosure, there is provided a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the above method.

[0012] According to one or more embodiments of the present disclosure, by acquiring state information of a set of movable units and feature information of a set of agents in a target environment, the influence degree of each individual movable unit and individual agent on each grid of the target environment is determined, so as to obtain a belief value effectively representing the existence probability of the movable unit in each grid. The belief value aggregates the effective information load of both the set of movable units as the observed object and the set of agents as the observer, greatly retaining the spatial information of the target environment. At the same time, in the process of constructing the belief graph, the information of each agent in the set of agents is fused into the determined belief value, so that the agents can cooperate in combat by means of the belief graph constructed according to the belief value, and the exploration efficiency of the target environment is improved.

[0013] These and other aspects of the present disclosure will become apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0014] In the following description of example embodiments in conjunction with the attached drawings, more details, features and advantages of the present disclosure are disclosed, in which:

[0015] Figure 1 is a schematic diagram illustrating an example system in which various methods described herein can be implemented according to example embodiments;

[0016] Figure 2 is a flowchart illustrating a belief graph construction method of a target environment according to example embodiments;

[0017] Figure 3 is a schematic diagram illustrating an example relationship among movable units, grids and agents according to example embodiments;

[0018] Figure 4 is a flowchart illustrating a process of determining a first influence degree representing the influence of each movable unit on a grid from the perspective of an agent according to example embodiments;

[0019] Figure 5 is a flowchart illustrating a process of determining a second influence degree representing the influence of each agent on a grid according to example embodiments;

[0020] Figure 6 is a flowchart illustrating a process of obtaining a set of belief values of a target environment according to example embodiments;

[0021] Figure 7 is a flowchart illustrating an agent training method according to example embodiments;

[0022] Figure 8is a schematic block diagram illustrating a belief map construction apparatus of a target environment according to an example embodiment;

[0023] Figure 9 is a schematic block diagram illustrating an agent training apparatus according to an example embodiment;

[0024] Figure 10 is a simulation example diagram illustrating an agent tracking a movable unit in a target environment according to an example embodiment;

[0025] Figure 11 is a block diagram illustrating an example computer device that can be applied to an example embodiment. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, in which various specific details are set forth to assist in a thorough understanding of the embodiments. It will be understood by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Also, the description is made in the order of the following description for clarity and conciseness, and the description of known functions and constructions can be omitted.

[0027] In the present disclosure, the terms "first", "second", and the like are used to describe various elements only for the purpose of distinguishing one element from another, and the terms are not intended to limit the positions, sequence, or importance of the elements. In some examples, a first element and a second element can refer to the same instance of the element, and in some cases, they can refer to different instances of the element based on the context of the description.

[0028] The terms used in the description of various described examples in the present disclosure are only for the purpose of describing particular examples and are not intended to be limiting. Unless specifically defined otherwise, an element that is a singular can be plural and vice versa. Further, the term "and / or" used in the present disclosure encompasses any and all possible combinations of the listed items.

[0029] With the development of technology, modern warfare is gradually moving towards electronicization and informatization. How to obtain accurate and complete information of the combat environment becomes particularly important. In this regard, environment modeling has become one of the key research works. Efficient representation of the distribution of movable units is an important part of environment modeling. Traditional environment modeling methods are usually characterized by vectorization. From the perspective of the observer, the vectorization representation method loses the spatial information of the environment. In contrast, the grid representation can retain the spatial information of the environment, but how to make the grid accurately and efficiently reflect the distribution of movable units and further promote cooperative combat has become a problem to be solved.

[0030] Based on this, the disclosure proposes a belief map construction method of a target environment. By acquiring state information of a movable unit set and feature information of an agent set in the target environment, the influence degree of each grid of the target environment by individual movable units and individual agents is determined, so that the belief value effectively representing the existence probability of the movable units in each grid can be obtained. The belief value aggregates the effective information load of both the movable unit set as the observed object and the agent set as the observer, greatly retaining the spatial information of the target environment. At the same time, in the construction process of the belief map, the information of each agent in the agent set is fused into the determined belief value, so that the agents can cooperate in combat by means of the belief map constructed according to the belief value, and the exploration efficiency of the target environment is improved.

[0031] Exemplary embodiments of the disclosure are described in detail below with reference to the accompanying drawings.

[0032] Figure 1 is a schematic diagram illustrating an example system 100 in which various methods described herein can be implemented, in accordance with an exemplary embodiment.

[0033] With reference to Figure 1 The system 100 includes a client device 110, a server 120, and a network 130 communicatively coupling the client device 110 and the server 120.

[0034] The client device 110 includes a display 114 and a client application (APP) 112 displayable via the display 114. The client application 112 can be an application program that needs to be downloaded and installed before running or a lite app as a lightweight application program. In the case where the client application 112 is an application program that needs to be downloaded and installed before running, the client application 112 can be pre-installed on the client device 110 and activated. In the case where the client application 112 is a lite app, the user 102 can directly run the client application 112 on the client device 110 without installing the client application 112 by searching for the client application 112 (e.g., by the name of the client application 112, etc.) in a host application or scanning a graphic code (e.g., a bar code, a two-dimensional code, etc.) of the client application 112, etc. In some embodiments, the client device 110 can be any type of mobile computer device, including a mobile computer, a mobile phone, a wearable computer device (e.g., a smart watch, a head-mounted device, including smart glasses, etc.), or other types of mobile devices. In some embodiments, the client device 110 can alternatively be a stationary computer device, such as a desktop computer, a server computer, or other types of stationary computer devices.

[0035] The server 120 is typically a server deployed by an Internet service provider (ISP) or an Internet content provider (ICP). The server 120 can represent a single server, a cluster of multiple servers, a distributed system, or a cloud server providing a basic cloud service such as cloud database, cloud computing, cloud storage, cloud communication. It will be understood that, although Figure 1 The server 120 is shown in FIG. 1 communicating with only one client device 110, but the server 120 can provide background services for multiple client devices at the same time.

[0036] Examples of the network 130 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), and / or a combination of communication networks such as the Internet. The network 130 can be a wired or wireless network. In some embodiments, data exchanged over the network 130 is handled using technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc. Furthermore, encryption technologies such as secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet Protocol security (IPsec), etc. can be used to encrypt all or some links. In some embodiments, custom and / or private data communication technologies can also be used instead of or in addition to the above-mentioned data communication technologies.

[0037] For the purpose of the embodiments of the present disclosure, in Figure 1 In an example, the client application 112 can be a simulation campaign design program that can provide various functions for belief graph construction and / or agent training of a target environment, e.g., target environment setting, movable unit set setting, agent set setting, and / or machine learning model configuration, etc. Correspondingly, the server 120 can be a server used with the simulation campaign design program. The server 120 can provide services related to belief graph construction and / or agent training of a target environment to the client application 112 running in the client device 110. Alternatively, the localized services can also be provided by the client application 112 running in the client device 110. As a non-limiting example, the methods described below can be performed by the system 100 of Figure 1

[0038] Figure 2 is a flowchart illustrating a belief graph construction method 200 of a target environment according to an example embodiment. The method 200 can be performed at a client device (e.g., the client device 110 shown in Figure 1 The execution subject of each step of the method 200 can be the client device 110 shown in Figure 1 In some embodiments, the method 200 can be performed at a server (e.g., the server 120 shown in Figure 1 ​The method 200 is executed at server 120 (as shown in the figure). In some embodiments, the method 200 may be executed in combination by a client device (e.g., client device 110) and a server (e.g., server 120).

[0039] refer to Figure 2 The method for constructing a belief map of the target environment 200 includes the following steps.

[0040] Step S210: For each agent in the set of agents in the target environment, obtain the state information of the set of mobile units in the target environment. The state information includes the current state estimate and associated uncertainty of each mobile unit in the set of mobile units from the perspective of the agent.

[0041] Step S220: For each grid cell in the target environment, determine the first degree of influence of each mobile unit on the grid cell from the perspective of the agent based on the state information of the set of mobile units, so as to obtain the first multiple first degrees of influence.

[0042] Step S230: Obtain feature information of the set of agents, including the current position and detection parameters of each agent in the set of agents;

[0043] Step S240: For each grid cell in the target environment, determine the second degree of influence representing the influence of each agent on the grid cell based on the feature information of the agent set, so as to obtain a second plurality of second degrees of influence.

[0044] Step S250: Based on the first plurality of first influence degrees and the second plurality of second influence degrees of each grid cell in the target environment, determine belief values ​​indicating the probability of the existence of movable units within the grid cells, to obtain a set of belief values ​​for the target environment; and

[0045] Step S260: Construct a belief map of the target environment based on the belief value set. The belief map includes color blocks that are associated with each grid of the target environment, wherein there is a correspondence between the belief value of the grid associated with the color block and the color of the color block.

[0046] The steps of method 200 are described in detail below.

[0047] In step S210, the target environment can be a real environment (e.g., a real battlefield environment), a simulated environment (e.g., a virtual battlefield environment), or a hybrid environment (e.g., including both a real battlefield environment and a virtual battlefield environment as components), and so on. As used herein, the term movable unit refers to a target that is capable of moving within the target environment (e.g., moving in a random manner, moving in a formation manner, and so on). In some embodiments, the set of movable units can include one or more movable units. In some embodiments, the movement of one or more movable units in the set of movable units within the target environment can not be continuously observable by some or all of the observers. In some embodiments, an individual movable unit can have its own state information. In some embodiments, some of the movable units in the set of movable units can have substantially the same state information, which means that these movable units can be in close proximity to each other and move in a formation manner, and so on.

[0048] Generally, for an individual combatant, the environment is partially observable. In order to be able to guide the actions of the combatant, a state estimate of the environment is needed, and in the active target tracking problem, the state of the target (e.g., a movable unit) affects the state of the environment in which it is located and thus needs to be estimated. Therefore, obtaining a state estimate of the target and the associated uncertainty is of great significance. In this document, a combatant can be an agent as described below.

[0049] In some embodiments, the state information of an individual movable unit can include a current state estimate and an associated uncertainty, which refers to the extent to which the state estimate result cannot be certain due to the existence of uncontrollable factors (e.g., due to the observer itself or introduced during the observation process, and so on), characterizing the range in which the estimated state can occur. It is noted here that as part of the state information, the uncertainty is necessary for determining the first impact degree and agent training set forth below. As described above, the movable units in the present disclosure can not be continuously observable by the observers, and thus the estimation of the state of the movable units by the observers is not certain, and the uncertainty can be advantageously used to characterize the extent to which the current state estimate of the movable units cannot be certain due to the existence of uncontrollable factors such as the observer itself (e.g., the observation ability of the observer is limited) or introduced during the observation process (e.g., sensor noise).

[0050] In step S220, for each grid of the target environment, the first influence degree of each movable unit in the given agent's perspective on the influence caused by the movable unit on the grid can be determined according to the state information of the movable unit. Thus, for each grid, the first influence degrees of the set of movable units in the given agent's perspective on the grid can include the first influence degree of each movable unit in the set of movable units on the grid. As used herein, the term first influence degree refers to the degree of influence of a movable unit on a grid in the given agent's perspective, and in particular, refers to the degree of influence of a movable unit on the belief value corresponding to each grid in the belief map of the target environment set forth below, which is derived from the relationship between the physical properties of the movable unit itself and the grid within the target environment, which is based on the observations of the given agent. Thus, it can be appreciated that the degree of influence of the same movable unit on the same grid in different agent's perspectives can be different, depending on the spatial relationship between the agent and the grid, the spatial relationship between the agent and the movable unit, the detection capability of the agent itself, or a combination thereof, and the like. The example relationship between a movable unit and a grid in the given agent's perspective within the target environment will be explained below with reference to FIG. 3. Figure 3

[0051] In some embodiments, for each grid, the first influence degree of each movable unit in the given agent's perspective on the grid can be determined, thereby obtaining a first plurality of first influence degrees. Specifically, assuming that the set of agents in the target environment contains M agents, and the set of movable units contains N movable units. For any grid c and any agent m, the total N first influence degrees of each movable unit in the perspective of the agent m on the grid c can be determined first, and thus the total M sets of first influence degrees of each movable unit in the perspective of each agent on the grid c can be further determined, where each set of first influence degrees includes N first influence degrees. That is, in the above example, the number of the first plurality of first influence degrees can be M x N. It is noted that the above example is merely illustrative, and is not intended to limit the scope of the present disclosure.

[0052] ​As used herein, the term agent can have multiple meanings. First, an agent is an important concept in the field of artificial intelligence, and any independent entity capable of thinking and interacting with the environment can be abstracted as an agent. According to the relevant definition, an agent is an entity residing in an environment, which can interpret the data obtained from the environment reflecting the events occurring in the environment, and perform actions that have an impact on the environment. In this definition, the agent is regarded as an entity that "survives" in the environment. Second, in the field of military or exploration, an agent can refer to an entity or sensor device equipped with various sensors to enable exploration of the environment, which can act as an observer or combatant in the combat environment relative to the movable unit, and can track targets in the combat environment. Therefore, the term agent as used herein can not only be a software element in the field of artificial intelligence, but also a hardware with an entity (such as a sensor device, a combat entity with exploration capability, etc.) or a combination of software and hardware, etc.

[0053] In step S230, the feature information of the agent set can be obtained, for example, from the agent set setting information in the simulation combat experiment design program. In some embodiments, the agent set can include one or more agents. In some embodiments, an individual agent can have its own feature information. In some embodiments, some agents in the agent set can have substantially the same feature information, which means that these agents can have substantially the same physical properties with respect to the target environment (for example, the field of view angle, the sight distance, etc. of the agent).

[0054] In some embodiments, the feature information of an individual agent can include the current position and the detection parameter of the agent, and the detection parameter can represent the detection capability of the agent with respect to the target environment, for example, the field of view angle, the sight distance, etc. It should be noted here that the detection capability of an individual agent with respect to the target environment can depend on multiple factors, and the detection capability of the agent can not change proportionally with respect to any of these factors. For example, when a certain grid of the target environment is outside the maximum sight distance of the agent, no matter how the distance between the grid and the agent changes, the agent cannot effectively detect the grid.

[0055] In step S240, for each grid of the target environment, a second degree of influence of each agent on the influence caused by the agent on the grid can be determined according to the characteristic information of the agent. Thus, for each grid, the second degrees of influence of the set of agents on the grid can include the second degrees of influence of each agent in the set of agents on the grid. As used herein, the term second degree of influence refers to the degree of influence of an agent on a grid, and in particular, to the degree of influence of an agent on the value of the belief value corresponding to each grid in the belief map of the target environment as set out below, which is derived from the relationship between the agent and the grid within the target environment. Example relationships between an agent and a grid within the target environment will also be explained below with reference to Figure 3

[0056] As mentioned above, the state of a movable unit influences the state of the target environment in which it is located, that is, the state of a movable unit has an influence on each grid of the target environment. Similarly, an agent as an observer also has an influence on each grid of the target environment. Specifically, assume that there is actually a movable unit in a given grid, and there are two agents in the target environment, which are at different distances from the grid (i.e., one is close to the grid, and the other is far from the grid) but have the same characteristic information. It can be understood that, in the case where the grid is located within the field of view of both agents, the detection result of the agent close to the grid can have a relatively high credibility, and the detection result of the agent far from the grid can have a relatively low credibility. It can be seen that the "influence" of different agents on the same grid in the target environment is obviously different. Of course, the above example can further assume that the two agents have different characteristic information. For example, the agent close to the grid has a relatively good detection capability, and the agent far from the grid has a relatively poor detection capability, then the detection result of the agent close to the grid can be given a higher credibility, and so on. Thus, in this sense, the second degree of influence can represent the degree of influence of an agent as an observer on a grid of the target environment, thereby facilitating the reflection of the differences in the influence of individual agents in the set of agents on the grid.

[0057] ​In some embodiments, for each grid, a second influence degree of each agent on the grid can be determined, thereby obtaining a second plurality of second influence degrees. Specifically, assuming that the set of agents in the target environment contains M agents. For any grid c and any agent m, a second influence degree of the agent m on the grid c can be first determined, and thus a total of M second influence degrees of each agent on the grid c can be further determined. That is, in the above example, the number of the second plurality of second influence degrees can be M. It is to be noted that the above example is merely illustrative, and is not intended to limit the scope of the present disclosure.

[0058] In step S250, for one grid, the belief value corresponding to the grid can indicate the possibility of the presence of movable units (e.g., one or more movable units) in the grid. In some embodiments, the belief value corresponding to each grid can be a numerical value. In other embodiments, the belief value corresponding to each grid can be a plurality of numerical values (e.g., represented as a vector or an array, etc.), each of which is due to the detection of the target environment by a different agent. In the first case where the belief value corresponding to each grid is a numerical value, the resulting belief map of the target environment can take the form of a matrix of color blocks corresponding to the spatial positions of the grids of the target environment, and what is obtained is a global belief map obtained by fusing the local belief maps obtained by the detection of the target environment by a plurality of agents respectively. In the second case where the belief value corresponding to each grid is a plurality of numerical values, the resulting belief map of the target environment can be a plurality of local belief maps obtained by the detection of the target environment by a plurality of agents respectively, and these local belief maps can be fused to obtain a global belief map.

[0059] In step S260, the obtained belief map can be visually represented as including color blocks respectively associated with each grid of the target environment. In some embodiments, there can be a correspondence between the belief value of the grid associated with the color block and the color of the color block, so as to visually represent the possibility of the presence of movable units in different grids. For example, the higher the belief value of the color block, the darker the color of the color block, and so on. As a result, the constructed belief map of the target environment can be subsequently image-processed using image processing techniques to obtain a feature vector capable of efficiently characterizing the target environment for further processing.

[0060] According to embodiments of this disclosure, by acquiring the state information of the set of mobile units and the feature information of the set of agents in the target environment, the influence of individual mobile units and individual agents on each grid of the target environment is determined. This allows for the acquisition of belief values ​​that effectively characterize the probability of the existence of mobile units in each grid. The belief values ​​aggregate the effective information payloads of both the set of mobile units as the observed object and the set of agents as the observers, thus preserving the spatial information of the target environment to a great extent. Simultaneously, during the construction of the belief graph, the information of each agent in the set of agents is integrated into the determined belief values, enabling agents to cooperate in combat using the belief graph constructed based on the belief values, thereby improving the efficiency of exploring the target environment.

[0061] Figure 3 This is a schematic diagram illustrating an example relationship between a movable unit, a grid, and an agent according to an exemplary embodiment. Figure 3 As shown, the target environment 300 may include a mobile unit (target) 301 and an agent (agent) 302. For illustrative purposes, only one mobile unit 301 and one agent 302 are shown; however, it should be understood that any number of mobile units 301 and / or any suitable number of agents 302 may exist within the target environment 300, and the number of mobile units 301 and the number of agents 302 need not be equal. This disclosure does not impose any limitations in this regard. In some embodiments, the target environment 300 is a gridded environment. As shown, one of the multiple grids of the target environment 300, cell 305, is shown. Although Figure 3 The target environment 300 is illustrated in a chessboard layout, but it should be understood that the target environment 300 can be rasterized in any other suitable manner, such as a honeycomb layout, a diamond grid layout, a triangular grid layout, etc. This disclosure does not impose any limitations in this regard.

[0062] As shown in the figure, and as an example rather than a limitation, the movable unit 301 may have a state estimate in a first coordinate system (e.g., a Cartesian coordinate system). (e.g., position estimation) and corresponding state estimation in a second coordinate system (e.g., polar coordinate system). ,in, This can represent a vector pointing from the movable unit 301 to the grid 305. Furthermore, the physical properties associated with the movable unit 301 may also include... and ,in, It can represent the velocity vector of a movable unit 301, while This can represent subtracting a vector from a full circle. with vector The angle obtained after forming the angle.

[0063] In the example, This can represent the state estimation of the movable unit 301 in the second coordinate system. The associated uncertainties. As a result, we can obtain the second coordinate system with... The mean and with The probability density function of a normal distribution with covariance can be denoted as: Accordingly, the movable unit 301 may have a probability density function in the first coordinate system, the mean and covariance of which will be described below.

[0064] In the example, a similarity function can also be constructed between the movable unit 301 and the grid 305 to serve as a component of the first influence of the movable unit 301 on the grid 305. This is intended as an example, not a limitation, of the similarity function. It can be represented as follows:

[0065] (1)

[0066] in, This represents the state estimation of the movable unit 301 in the first coordinate system. The derivative or difference, This represents the maximum speed of the mobile unit 301, which can be configured and set in the simulation combat experiment design program, or determined based on the mobile unit's attributes and / or historical data. In the example, absolute value and The ratio of the absolute values ​​can have a range from 0 to 1. The cosine similarity is calculated by taking the cosine of the angle between two vectors, which is the ratio of the dot product of the two vectors to the product of their magnitudes. The value ranges from -1 to 1. Therefore, in equation (1) above... The range of values ​​can be from Up to 1.

[0067] It should be understood that equation (1) above is merely illustrative and not intended to limit the scope of this disclosure. Any other suitable similarity function can be constructed based on the state estimate of the movable unit and the associated uncertainty, as well as other physical properties associated with the movable unit, to characterize the similarity between the movable unit and the raster in some sense, such that the value of the similarity can be any subset of the interval 0 to 1.

[0068] As an example rather than a limitation, the first influence q can be represented as follows:

[0069] (2)

[0070] Here, the first degree of influence q is represented as , to indicate in the first i From the perspective of an individual agent, movable units j The degree of influence on grid c. In the example, the first degree of influence can be represented by a probability density function. Similarity function The product of. It should be noted that in this disclosure, each agent can perform state estimation for all mobile units in the target environment. It is understood that the above equation (2) is merely illustrative and not intended to limit the scope of this disclosure.

[0071] Figure 4 This is a flowchart illustrating a process 400 of determining a first degree of influence representing the effect of each movable unit on a grid from the perspective of an agent, according to an exemplary embodiment.

[0072] like Figure 4 As shown, process 400 includes:

[0073] Step S2201: For each agent, determine the first probability density of the mobile unit based on the current state estimate and associated uncertainty of each mobile unit to obtain multiple first probability densities;

[0074] Step S2202: Perform coordinate transformation on the multiple first probability densities to obtain multiple second probability densities;

[0075] Step S2203: Determine the spatial similarity measure of each movable unit relative to the grid based on the state information, and obtain multiple spatial similarity measures;

[0076] Step S2204: Determine the first influence degree representing the influence of the movable unit on the grid based on the second probability density and spatial similarity metric of each movable unit, and obtain multiple first influence degrees of the grid from the perspective of each agent to obtain the first multiple first influence degrees.

[0077] In step S2202, the first probability density in the first coordinate system is transformed to obtain the second probability density in the second coordinate system in order to facilitate the characterization of the physical properties of the movable unit itself and the relationship between the grid in the target environment.

[0078] In some embodiments, the current state estimation and associated uncertainty of the movable units can be obtained by filtering approaches. In an example, the current state estimation and associated uncertainty of the movable units in a first coordinate system (e.g., Cartesian coordinate system) can be obtained based on Kalman filtering approach. Of course, the present disclosure is not limited to Kalman filtering, and any other suitable filtering approach can be selected by those skilled in the art to obtain the state information of the set of movable units as needed. In Kalman filtering approach, the state estimation is divided into two steps, i.e., prediction and update. At each time step, the prediction is formulated as follows:

[0079] (3)

[0080] (4)

[0081] The above equation (3) estimates the state of the i-th movable unit, where represents the state transition matrix of the movable unit, represents the control matrix of the movable unit, represents the input control quantity, and represents the state estimation of the movable unit at time and The above equation (4) estimates the uncertainty of the movable unit, where and is the covariance matrix of the movable unit at time and is the covariance matrix of the process noise. In an example, the control matrix may be a unit matrix in the simplest form. In an example, the input control quantity may be used as a control variable of the pre-set dynamic equation of the movable unit to control the moving behavior of the movable unit, which is particularly useful in the use case of simulating the active tracking of the movable unit by the agent in the target environment. At each time, the state prediction of the movable unit is needed. Then, for each movable unit, if the agent obtains the observation of the movable unit, the movable unit will enter the update step, and if the corresponding observation is not obtained, the movable unit will not enter the update step, and the formula of the update step is as follows:

[0082] (5)

[0083] (6)

[0084] (7)

[0085] ​​

[0086] The above equation (5) estimates the Kalman gain, where represents the measurement matrix, represents the covariance matrix of the measurement noise. The above equation (6) corrects the state of the movable unit using the Kalman gain and the observation value of the target . The above equation (7) corrects the uncertainty of the movable unit. It is understood that other means other than filtering can be taken to obtain the state information of the movable unit set, such as using one or more of data-driven, sequence neural network, uncertainty quantification, LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), etc. to obtain the current state estimation and the associated uncertainty of the movable unit, respectively, without disclosing any limitation thereto.

[0087] Referring back to Figure 3 , by way of example and not limitation, the characteristic information of the agent 302 can include a maximum detection distance (i.e., a look distance of the agent) and a maximum detection field of view half angle (i.e., half of a maximum detection field of view angle). As used herein, the term maximum detection field of view angle corresponds to a maximum spatial angle enclosed by the boundaries of the field of view of the agent, while the maximum detection field of view half angle corresponds to half of the maximum spatial angle. As shown, for the agent i , the characteristic information may include the maximum detection distance (shown as ) and the maximum detection field of view half angle (shown as ). Further, the angular bisector (plane) of the maximum detection field of view angle is shown as , and has a directivity. It is noted that the characteristic information i of the agent and the angular bisector (plane) of the maximum detection field of view angle are described as functions of time t, but this does not necessarily mean that the relevant quantities change with time. For example, the maximum detection field of view half angle of the agent can remain unchanged with time t, but as the position of the agent i changes with each time step, the directivity also changes, and thus the angular bisector (plane) is denoted as a quantity that is a function of time t. As shown, the distance (line) between the agent 302 and the grid 305 is shown as , and the angle that the line makes with is shown as .

[0088] By way of example and not limitation, the second impact degree p can be obtained as represented by:

[0089] (8)

[0090] wherein the second influence degree p is represented as to represent the influence degree of the i first agent on the cell c, (·) is a sigmoid function. Thus, when it is determined that the cell is within the detection range of the agent according to the current position of the agent and the detection parameter, the second influence degree can be set to a determined value (e.g. 1), and when it is determined that the cell is outside the detection range of the agent, the influence degree of the agent on the cell when the cell is outside the detection range of the agent can be quantified by using the properties of the special function. Of course, the above formula (8) is only illustrative and is not intended to limit the scope of the present disclosure. The present disclosure is not limited to the sigmoid function, and any other suitable function or variant thereof can be selected by those skilled in the art as needed to be used as a replacement for the piecewise function of the above formula (8).

[0091] Figure 5 is a flowchart illustrating a process 500 of determining the second influence degree representing the influence of each agent on the cell according to an exemplary embodiment.

[0092] As shown in Figure 5 , the process 500 includes, for each agent in the set of agents:

[0093] Step S2401, determining whether the cell is within the detection range of the agent according to the current position of the agent and the detection parameter;

[0094] Step S2402a, in response to determining that the cell is within the detection range of the agent, setting the second influence degree to a determined value;

[0095] Step S2402b, in response to determining that the cell is outside the detection range of the agent, obtaining spatial relative position information of the agent relative to the cell, and inputting the spatial relative position information and the detection parameter of the agent into a target activation function to obtain a function value output by the target activation function as the second influence degree, wherein the maximum function value output by the target activation function is equal to the determined value.

[0096] Referring back to Figure 3 , after obtaining the first plurality of first influence degrees of the cell 305 based on the relationship between the movable unit 301 and the cell 305, and after obtaining the second plurality of second influence degrees of the cell 305 based on the relationship between the agent 302 and the cell 305, the belief value of the cell 305 can be determined to reflect the possibility of the existence of a movable unit within the cell 305.

[0097] As an example but not limitation, the belief value of the cell represented as :

[0098] (9)

[0099] It should be noted that equation (9) above applies to any grid c, where, according to the above, the given agent is first determined. i The first degree of influence of each movable unit on the grid from the perspective of (i.e., , j= 1 ,…,n These first influence values ​​are respectively associated with a given agent. i The second degree of influence of the raster (i.e., Multiply by , and obtain the given intelligent agent. i The degree of integration of multiple grids from different perspectives.

[0100] As used in this article, the term integration influence refers to the degree of influence of a given agent. i The degree of influence of movable units on the grid from the perspective of a given agent i The degree of influence of the agent on the grid is integrated to represent the influence of both the agent and the mobile unit on the grid.

[0101] Then, the various intelligent agents i ( i= 1 ,…,m The average influence of a grid is obtained by averaging multiple integrated influence values ​​of the grid from the perspective of an agent (e.g., arithmetic mean, weighted average, etc.). Specifically, for an agent... i First determine the expression ( j= 1 ,…,n Then, iterate through and determine the aforementioned expressions of all agents, summing them to obtain... ( i= 1 ,…,m ; j= 1 ,…,n Finally, an average is calculated to make the belief value... It falls within the range of 0 to 1. It is understood that equation (9) above is merely illustrative and not intended to limit the scope of this disclosure. Those skilled in the art can employ any other suitable averaging algorithm as needed.

[0102] Figure 6 This is a flowchart illustrating a process 600 for obtaining a set of belief values ​​for a target environment according to an exemplary embodiment.

[0103] like Figure 6 As shown, process 600 includes:

[0104] Step S2501, for each agent, integrating the plurality of first influence degrees of the grid in the agent's perspective and the second influence degree of the influence of the agent on the grid, to obtain a plurality of integrated influence degrees of the grid in the agent's perspective;

[0105] Step S2502, averaging the plurality of integrated influence degrees of the grid in each agent's perspective to obtain an average influence degree of the grid;

[0106] Step S2503, determining the belief value of each grid according to the average influence degree of each grid to obtain a belief value set of the target environment.

[0107] In step S2503, the average influence degree of each grid can be determined as the belief value of each grid. However, when the belief value corresponding to each grid is a plurality of values (for example, represented as a vector or an array, etc.), the plurality of values can be averaged (for example, arithmetic average, or weighted average (such as the value associated with the agent with strong detection ability detecting the target environment can be given a larger weight), etc.) to obtain the belief value of the grid.

[0108] Figure 7 is a flowchart illustrating an agent training method 700 according to an example embodiment. The method 700 can be executed at a client device (for example, the client device 110 shown in Figure 1 ), that is, the execution subject of each step of the method 700 can be the client device 110 shown in Figure 1 . In some embodiments, the method 700 can be executed at a server (for example, the server 120 shown in Figure 1 ). In some embodiments, the method 700 can be executed by a combination of a client device (for example, the client device 110) and a server (for example, the server 120).

[0109] Referring to Figure 7 , the agent training method 700 includes the following steps.

[0110] In the sample collection phase S71, the following operations are performed:

[0111] Step S711, for the current time step and the next time step, constructing a corresponding belief graph of the target environment according to any of the above methods;

[0112] Step S712, inputting the corresponding belief graph into the multi-layer convolutional neural network to obtain a corresponding plurality of sub-sampling feature vectors output by the multi-layer convolutional neural network;

[0113] Step S713, using an attention mechanism to determine a corresponding fusion feature vector as the current state and the next state of the target environment according to the corresponding plurality of sub-sampling feature vectors;

[0114] Step S714, interacting the selected policy of the target reinforcement learning model with the target environment at the current time step to obtain an action to be taken by the agent set at the current state, the action causing each agent in the agent set to move from a respective current position at the current time step to a respective position at a next time step;

[0115] Step S715, storing the current state, the action to be taken by the agent set at the current state, a reward obtained by the agent set taking the action at the current state, and the next state as a four-tuple training sample into the experience pool, wherein a number of four-tuple training samples required for one round is denoted as a first sample number; and

[0116] Step S716, in response to a number of newly stored four-tuple training samples in the experience pool reaching a preset multiple of the first sample number, interrupting the operation of the current sample collection stage and entering a model training stage; or

[0117] Step S717, in response to the number of newly stored four-tuple training samples in the experience pool not reaching the preset multiple of the first sample number, continuing the above operation,

[0118] In the model training stage S72, the following operations are performed:

[0119] Step S721, updating the policy of the reinforcement learning model using a batch of four-tuple training samples selected from the experience pool;

[0120] Step S722, taking the updated policy of the reinforcement learning model as a policy trained by the reinforcement learning model at the current model training stage; and

[0121] Step S723, comparing the policy trained at the current model training stage with one or more policies trained at previous model training stages to determine the selected policy of the target reinforcement learning model at a next sample collection stage.

[0122] The method 700 described above extracts feature maps of different receptive fields by a convolutional neural network from the constructed belief graph, and then fuses the features by an attention mechanism to obtain multi-receptive-field environmental features. In this way, the understanding of the environment by the agent is improved by using belief graphs under different receptive fields.

[0123] Since the training of the reinforcement learning model is unstable, the policy of the reinforcement learning model is updated at each update frequency (i.e., at each preset multiple of the first number of samples required for training per episode), thereby generating a new reinforcement learning model. The optimization result of the objective function obtained in the current model training stage (e.g., the function value after gradient descent) can be compared with the optimization result of the objective function obtained in the previous model training stage, so as to determine the reinforcement learning model strategy that is optimal as of the current model training stage. As can be seen, the two-stage (i.e., sample collection stage and model training stage) agent training method of method 700 can be used by the user to flexibly select a suitable reinforcement model strategy without having to be subject to the limitation of the objective function optimization falling into a local extremum or a longer number of iterations of episodes.

[0124] In some embodiments, the reward obtained by the set of agents taking the action in the current state can be represented as:

[0125] (10)

[0126] where det(·) represents the value of the determinant of a square matrix (e.g., a covariance matrix).

[0127] In some embodiments, a return function can be constructed on the basis of the reward function, for example, based on a time-first average, i.e., assigning a lower weight to a reward that is slightly earlier in time and a higher weight to a reward that is more recent in time. Then, a target function for training the reinforcement learning model can be constructed based on the return function.

[0128] In some embodiments, the experience pool can be hot-started, i.e., the initial training samples in the experience pool are also obtained through the steps described in relation to the sample collection stage of method 700.

[0129] In some embodiments, at the beginning of each episode, the number of movable units in the set of movable units in the target environment and / or the number of agents in the set of agents can be changed. In this way, it can be expected that the trained reinforcement learning model is suitable for the active tracking task in the target environment including any number of movable units and / or agents.

[0130] Figure 8 is a schematic block diagram illustrating a belief graph construction apparatus 800 of a target environment according to an exemplary embodiment. As Figure 8As shown, the device 800 includes: a first acquisition module 810, configured to acquire state information of a set of movable units in a target environment for each agent in the set of agents, the state information including the current state estimate and associated uncertainty of each movable unit in the set of movable units from the agent's perspective; a first determination module 820, configured to determine a first influence degree representing the influence of each movable unit on the grid from the agent's perspective on each grid cell in the target environment, based on the state information of the set of movable units, to obtain a first plurality of first influence degrees; and a second acquisition module 830, configured to acquire feature information of the set of agents, the feature information including the current position and position of each agent in the set of agents. The detection parameters; a second determining module 840, configured to determine a second degree of influence representing the influence of each agent on the grid for each grid of the target environment based on the feature information of the agent set, so as to obtain a second plurality of second degrees of influence; a third determining module 850, configured to determine a belief value indicating the possibility of the existence of a movable unit in the grid based on a first plurality of first degrees of influence and a second plurality of second degrees of influence for each grid of the target environment, so as to obtain a set of belief values ​​for the target environment; and a construction module 860, configured to construct a belief map of the target environment based on the set of belief values, the belief map including color patches associated with each grid of the target environment, wherein there is a correspondence between the belief value of the grid associated with the color patch and the color of the color patch.

[0131] It should be understood that Figure 8 The various modules of the device 800 shown can be connected to the reference. Figure 2 The steps in method 200 described correspond to each other. Therefore, the operations, features, and advantages described above for method 200 also apply to device 800 and its included modules. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0132] Figure 9 This is a schematic block diagram illustrating an agent training device 900 according to an exemplary embodiment. Figure 9As shown, the device 900 includes: a sample acquisition module 91, comprising: a first unit 911, used to construct a corresponding belief map of the target environment for the current time step and the next time step according to any of the methods described above; a second unit 912, used to input the corresponding belief map into a multi-layer convolutional neural network to obtain corresponding multiple sub-sampled feature vectors output by the multi-layer convolutional neural network; a third unit 913, used to use an attention mechanism to determine corresponding fused feature vectors as the current state and the next state of the target environment based on the corresponding multiple sub-sampled feature vectors; a fourth unit 914, used to enable the selected policy of the target reinforcement learning model to interact with the target environment at the current time step to obtain the action to be taken by the agent set in the current state, the action causing each agent in the agent set to move from the corresponding current position at the current time step to the corresponding position at the next time step; and a fifth unit 915, used to use the current state, the action to be taken by the agent set in the current state, the reward obtained by the agent set in the current state, and the next state as... The training samples of quadruples are stored in the experience pool, wherein the number of training samples of quadruples required for one round is denoted as the first sample quantity; and the sixth unit 916 is used to interrupt the operation of the current sample collection phase and enter the model training phase in response to the number of newly stored training samples of quadruples in the experience pool reaching a preset multiple of the first sample quantity; and the seventh unit 917 is used to continue the above operation in response to the number of newly stored training samples of quadruples in the experience pool not reaching a preset multiple of the first sample quantity. The model training module 92 includes: the eighth unit 921, used to update the policy of the reinforcement learning model using a batch of training samples of quadruples selected from the experience pool; the ninth unit 922, used to use the updated policy of the reinforcement learning model as the policy obtained by the reinforcement learning model in the current model training phase; and the tenth unit 923, used to compare the policy obtained by the current model training phase with one or more policies obtained by the previous model training phase to determine the selected policy of the target reinforcement learning model in the next sample collection phase.

[0133] It should be understood that Figure 9 The various modules / units of the device 900 shown can be connected to the reference. Figure 7 The steps in method 700 described correspond to each other. Therefore, the operations, features, and advantages described above for method 700 also apply to apparatus 900 and its included modules. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0134] While certain functions have been discussed above with reference to particular modules, it should be noted that the functions of the various modules discussed herein can be split among multiple modules and / or at least some of the functions of multiple modules can be combined into a single module. A particular module discussed herein performing an action includes that particular module itself performing the action, or alternatively, that particular module invoking or otherwise accessing another component or module that performs the action (or performs the portion of the action that is relevant to that particular module). Thus, a particular module performing an action can include that particular module itself performing the action and / or another module accessing or otherwise performing the action.

[0135] It should also be understood that various techniques described herein can be described in the general context of software hardware elements, or program modules. The terms "software" and "program" are used herein to refer to any software hardware elements, or program modules, including one or more applications, operating system interfaces, libraries, routines, programs, components, modules, or the like. The various software hardware elements and / or program modules described herein can be implemented in software and / or firmware for execution by one or more processors such as the processor 1020. Figure 8-9 The various modules and / or units described above can be implemented in hardware or in hardware combined with software and / or firmware. For example, these modules and / or units can be implemented as computer program code / instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules and / or units can be implemented as hardware logic / circuitry. One or more of the modules and / or units can be implemented together in a System on Chip (SoC). The SoC can include an integrated circuit chip (which includes one or more of a processor (e.g., a Central Processing Unit (CPU), a microcontroller, a microprocessor, a Digital Signal Processor (DSP), etc.), a memory, one or more communication interfaces, and / or other circuitry), and can optionally execute received program code and / or include embedded firmware to perform functions.

[0136] Figure 10 FIG. 10 is a diagram illustrating a simulation example 1000 of an agent tracking a movable unit in a target environment, according to an example embodiment.

[0137] The experiment sets up opposing sides: the blue team uses multiple mobile sensors (i.e., agents) to detect the environment, with each sensor making independent decisions; the red team has multiple mobile units that move randomly or in formation within the environment. In the simulation, the red team's mobile units move randomly, while the blue team controls the actions of the mobile sensors through a control strategy. As shown in Figures (a) and (b), where the instance in Figure (a) is earlier in time than the instance in Figure (b), when two-unit formations exist, the agents tend to track high-value targets (i.e., two-unit formations with a larger number of mobile units). When the two-unit formations disband, as shown in Figure (c), the agents, guided by an updated strategy, switch to tracking individual discrete targets separately. The trajectories of the mobile units in Figures (a) to (c) are shown as dense and continuous coarse discrete points, while the tracking trajectories of the agents are shown as relatively sparse fine discrete points. Therefore, by using the belief graph construction method of the target environment disclosed herein, agents can cooperate in combat by using the belief graph constructed based on belief values, thereby improving the efficiency of exploring the target environment.

[0138] According to one aspect of this disclosure, a computer device is provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0139] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0140] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0141] In the following text, combined with Figure 11 Illustrative examples describing such computer devices, non-transitory computer-readable storage media, and computer program products.

[0142] Figure 11 An example configuration of a computer device 1100 that can be used to implement the methods described herein is shown. For example, Figure 1 The server 120 and / or client device 110 shown may include an architecture similar to computer device 1100. The aforementioned devices 800 and 900 may also be implemented wholly or at least partially by computer device 1100 or similar devices or systems.

[0143] The computer device 1100 can be various different types of devices. Examples of the computer device 1100 include, but are not limited to: a desktop computer, a server computer, a notebook or netbook computer, a mobile device (e.g., a tablet computer, a cellular or other wireless phone (e.g., a smart phone), a notepad computer, a mobile station), a wearable device (e.g., glasses, a watch), an entertainment device (e.g., an entertainment appliance, a set-top box, a game console), a television or other display device, an automobile computer, and so forth.

[0144] The computer device 1100 can include at least one processor 1102, memory 1104, communication interface(s) 1106, a display device 1108, other input / output (I / O) devices 1110, and one or more mass storage devices 1112, which can communicate with one another by way of a system bus 1114 or other appropriate connection, such as a wired or wireless communication connection.

[0145] The processor 1102 can be a single processing unit or a plurality of processing units, all of which can include single or multiple computing units or multiple cores. The processor 1102 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any devices that manipulate signals in accordance with operational instructions. Among other capabilities, the processor 1102 can be configured to fetch and execute computer-readable instructions stored in the memory 1104, the mass storage device 1112, or any other computer-readable medium, such as program code for an operating system 1116, program code for applications 1118, program code for other programs 1120, and so forth.

[0146] Memory 1104 and mass storage device 1112 are examples of computer readable storage media for storing instructions that are executed by processor 1102 to implement the various functions described above. By way of example, the memory 1104 can generally include both volatile memory and nonvolatile memory (e.g., RAM, ROM, etc.). In addition, mass storage device 1112 can generally include hard disk drives, solid state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CD or DVD), storage arrays, network attached storage, storage area networks, etc. Both memory 1104 and mass storage device 1112 can be collectively referred to herein as memory or computer readable storage media, and can be non-transitory media capable of storing computer readable, processor executable program instructions as computer program code that can be executed by processor 1102 as a particular machine configured to implement the operations and functions described in the examples herein. Multiple programs can be stored on the mass storage device 1112. These programs include an operating system 1116, one or more application programs 1118, other programs 1120, and program data 1122, and they can be loaded into memory 1104 for execution.

[0147] Although illustrated in Figure 11 the memory 1104 of computer device 1100, blocks 1116, 1118, 1120, and 1122, or portions thereof, can be implemented using any form of computer readable media that is accessible by the computer device 1100. As used herein, "computer readable media" includes both computer readable storage media and communication media.

[0148] Computer readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computer device. In contrast, communication media can embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transport mechanism. As defined herein, computer readable storage media does not include communication media.

[0149] One or more communication interfaces 1106 are used to exchange data to and from other devices, such as over a network, direct connection, or the like. Such communication interfaces can be one or more of: any type of network interface (e.g., network interface card (NIC)), wired or wireless (such as IEEE 802.11 wireless LAN (WLAN)) wireless interface, Worldwide Interoperability Microwave Access (Wi-MAX) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth TM interface, near field communication (NFC) interface, or the like. The communication interfaces 1106 can facilitate communication over a variety of network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, and the like. The communication interfaces 1106 can also provide communication with external storage (not shown), such as in a storage array, network attached storage, storage area network, or the like.

[0150] In some examples, a display device 1108, such as a monitor, can be included for displaying information and images to a user. Other I / O devices 1110 can be devices that receive various inputs from a user and provide various outputs to the user, and can include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and the like.

[0151] The technology described herein can be supported by these various configurations of the computer device 1100 and is not limited to the specific examples described herein. For example, the functionality can also be implemented all or in part through use of a distributed system, such as over a "cloud." Cloud includes and / or comprises a platform of resources. The platform abstracts underlying functionality of hardware (e.g., servers) and software resources of the cloud. Resources can include applications and / or data that can be utilized while a computing process is executed on a server that is remote to the computer device 1100. Resources can also include services provided over the Internet and / or over a subscriber network, such as a cellular or Wi-Fi network. The platform can abstract resources and functionality to connect the computer device 1100 with other computer devices. Accordingly, the implementation of functionality described herein can be distributed throughout the cloud. For example, functionality can be implemented in part on the computer device 1100 and in part by the platform that abstracts the functionality of the cloud.

[0152] While the disclosure has been illustrated and described in detail in the drawings and foregoing description, such illustration and description is to be considered illustrative or exemplary and not restrictive; the disclosure is not limited to the disclosed embodiments. Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed subject matter, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, the word "a" or "an" does not exclude a plurality, the term "multiple" means two or more, and the term "based on" means "based at least in part on." The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A method for constructing a belief map of a target environment, comprising: For each agent in the set of agents in the target environment, obtain the state information of the set of mobile units in the target environment. The target environment includes a real combat environment, a virtual combat environment, or a hybrid environment. The state information includes the current state estimate and associated uncertainty of each mobile unit in the set of mobile units from the perspective of the agent. For each grid cell in the target environment, a first degree of influence representing the impact of each movable unit on the grid cell from the agent's perspective is determined based on the state information of the movable unit set, to obtain a first plurality of first degrees of influence, including: For each agent, a first probability density of the mobile unit is determined based on the current state estimate and associated uncertainty of each mobile unit to obtain a plurality of first probability densities; The multiple first probability densities are transformed by coordinates to obtain multiple second probability densities; Based on the state information, a spatial similarity metric for each movable unit relative to the grid is determined, resulting in multiple spatial similarity metrics; Based on the second probability density of each movable unit and the spatial similarity metric, a first influence degree representing the influence of the movable unit on the grid is determined, and multiple first influence degrees of the grid from the perspective of each agent are obtained to obtain the first multiple first influence degrees. Obtain feature information of the set of intelligent agents, the feature information including the current position and detection parameters of each intelligent agent in the set of intelligent agents; For each grid cell in the target environment, a second influence degree representing the influence of each agent on the grid cell is determined based on the feature information of the agent set, to obtain a second plurality of second influence degrees, including: For each agent in the set of agents, Determine whether the grid is within the detection range of the agent based on the agent's current position and detection parameters; In response to determining that the grid is within the detection range of the agent, the second influence degree is set to a determined value; In response to determining that the grid is outside the detection range of the agent, the spatial relative position information of the agent relative to the grid is obtained, and the spatial relative position information and the detection parameters of the agent are input together into the target activation function to obtain the function value output by the target activation function as the second influence degree, wherein the maximum function value output by the target activation function is equal to the determined value; Based on the first plurality of first influence degrees and the second plurality of second influence degrees of each grid cell in the target environment, belief values ​​indicating the probability of the movable unit existing within the grid cell are determined to obtain a set of belief values ​​for the target environment; and A belief map of the target environment is constructed based on the set of belief values. The belief map includes color patches that are associated with each grid cell of the target environment, wherein there is a correspondence between the belief value of the grid cell associated with the color patch and the color of the color patch.

2. The method according to claim 1, wherein, The set of belief values ​​for the target environment includes: For each agent, multiple first influence degrees of the grid from the agent's perspective and a second influence degree of the agent's influence on the grid are integrated to obtain multiple integrated influence degrees of the grid from the agent's perspective. The average influence of the grid is obtained by averaging the multiple integrated influence values ​​of the grid from the perspective of each agent. The belief value of each grid is determined based on the average influence of each grid, thus obtaining the set of belief values ​​for the target environment.

3. A method for training an agent, comprising: During the sample collection phase, perform the following operations: For the current time step and the next time step, construct the corresponding belief map of the target environment according to any one of claims 1 to 2; The corresponding belief map is input into a multi-layer convolutional neural network to obtain the corresponding multiple sub-sampled feature vectors output by the multi-layer convolutional neural network; Using an attention mechanism, a corresponding fused feature vector is determined as the current state and the next state of the target environment based on the corresponding multiple sub-sampled feature vectors; At the current time step, the selected policy of the target reinforcement learning model interacts with the target environment to obtain the action to be taken by the agent set in the current state. The action causes each agent in the agent set to move from its corresponding current position at the current time step to its corresponding position at the next time step. The current state, the action to be taken by the agent set in the current state, the reward obtained by the agent set for taking the action in the current state, and the next state are stored as training samples for a quadruple in the experience pool, wherein the number of training samples for a quadruple required in one round is denoted as the first sample quantity; and In response to the number of newly added quadruple training samples in the experience pool reaching a preset multiple of the first sample number, the operation of the current sample collection phase is interrupted and the model training phase is entered; or If the number of newly added quadruplet training samples in the experience pool does not reach a preset multiple of the first sample number, the above operation continues. During the model training phase, perform the following operations: The strategy of updating the reinforcement learning model is to use a batch of training samples of quadruples selected from the experience pool; The updated policy of the reinforcement learning model is used as the policy obtained by training the reinforcement learning model in the current model training phase; and The strategy trained in the current model training phase is compared with one or more strategies trained in previous model training phases to determine the selected strategy of the target reinforcement learning model in the next sample collection phase.

4. The method according to claim 3, wherein, At the start of each round, the number of mobile units in the set of mobile units and / or the number of agents in the set of agents in the target environment can be changed.

5. A belief map construction device for a target environment, comprising: The first acquisition module is configured to acquire, for each agent in the set of agents in the target environment, the state information of the set of mobile units in the target environment, wherein the target environment includes a real combat environment, a virtual combat environment, or a hybrid environment, and the state information includes the current state estimate and associated uncertainty of each mobile unit in the set of mobile units from the perspective of the agent. The first determining module is configured to, for each grid cell in the target environment, determine a first influence degree representing the influence of each movable unit on the grid cell from the perspective of the agent, based on the state information of the movable unit set, to obtain a first plurality of first influence degrees, including: for each agent, determining a first probability density of the movable unit based on the current state estimate and associated uncertainty of each movable unit, to obtain a plurality of first probability densities; performing coordinate transformation on the plurality of first probability densities to obtain a plurality of second probability densities; determining a spatial similarity metric of each movable unit relative to the grid cell based on the state information, to obtain a plurality of spatial similarity metrics; determining the first influence degree representing the influence of the movable unit on the grid cell based on the second probability density of each movable unit and the spatial similarity metric, to obtain a plurality of first influence degrees of the grid cell from the perspective of each agent, to obtain the first plurality of first influence degrees; The second acquisition module is configured to acquire feature information of the intelligent agent set, the feature information including the current position and detection parameters of each intelligent agent in the intelligent agent set; The second determining module is configured to, for each grid cell in the target environment, determine a second degree of influence representing the influence of each agent on the grid cell based on the feature information of the agent set, to obtain a second plurality of second degrees of influence. This includes: for each agent in the agent set, determining whether the grid cell is within the agent's detection range based on the agent's current position and detection parameters; in response to determining that the grid cell is within the agent's detection range, setting the second degree of influence to a determined value; in response to determining that the grid cell is outside the agent's detection range, acquiring the spatial relative position information of the agent relative to the grid cell, and inputting the spatial relative position information and the agent's detection parameters together into a target activation function to obtain the function value output by the target activation function as the second degree of influence, wherein the maximum function value output by the target activation function is equal to the determined value. The third determining module is configured to determine, based on the first plurality of first influence degrees and the second plurality of second influence degrees of each grid cell in the target environment, a belief value indicating the possibility of the movable unit existing within the grid cell, to obtain a set of belief values ​​for the target environment; and The construction module is configured to construct a belief graph of the target environment based on the set of belief values. The belief graph includes color blocks that are associated with each grid of the target environment, wherein there is a correspondence between the belief value of the grid associated with the color block and the color of the color block.

6. An intelligent agent training device, comprising: The sample collection module includes: The first unit is used to construct a corresponding belief map of the target environment for the current time step and the next time step, according to the method of any one of claims 1 to 2. The second unit is used to input the corresponding belief map into a multi-layer convolutional neural network to obtain the corresponding multiple sub-sampled feature vectors output by the multi-layer convolutional neural network; The third unit is used to utilize an attention mechanism to determine the corresponding fused feature vector as the current state and the next state of the target environment based on the corresponding multiple sub-sampled feature vectors; The fourth unit is used to enable the selected policy of the target reinforcement learning model to interact with the target environment at the current time step, and obtain the action to be taken by the agent set in the current state. The action causes each agent in the agent set to move from the corresponding current position at the current time step to the corresponding position at the next time step. The fifth unit is used to store the current state, the action to be taken by the set of agents in the current state, the reward obtained by the set of agents for taking the action in the current state, and the next state as training samples for a quadruple into the experience pool, wherein the number of training samples for a quadruple required in one round is denoted as the first sample quantity; and The sixth unit is configured to interrupt the current sample acquisition phase and enter the model training phase in response to the number of newly added quadruple training samples in the experience pool reaching a preset multiple of the first sample number; and The seventh unit is configured to continue the above operation in response to the fact that the number of newly added quadruplet training samples in the experience pool does not reach a preset multiple of the number of the first samples. The model training module includes: The eighth unit is used to update the strategy of the reinforcement learning model using a batch of four-tuple training samples selected from the experience pool; The ninth unit is used to take the updated policy of the reinforcement learning model as the policy obtained by training the reinforcement learning model in the current model training phase; and The tenth unit is used to compare the policy trained in the current model training phase with one or more policies trained in the previous model training phase, and to determine the selected policy of the target reinforcement learning model in the next sample collection phase.

7. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores a computer program that, when executed by the at least one processor, implements the method according to any one of claims 1-4.

8. A non-transitory computer-readable storage medium storing a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-4.

Images