Generalized state representation method for game confrontation intelligent algorithm
By dividing agents into discrete meshes in a multi-agent environment and building feature matrix and observation matrix, the problem of insufficient generalization ability of generalization framework when the number of agents changes is solved, and efficient generalization of reinforcement learning models is achieved and adaptability.
Patent Information
- Application Number
- CN202510010992.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-03
- Publication Date
- 2025-02-07
- Estimated Expiration
- 2045-01-03
AI Technical Summary
The existing general decision-making framework is difficult to generalize effectively in a multiagent environment, especially when the number of agents changes, the dimension and structural changes of the state space pose challenges to the generalization ability of the decision-making model.
By obtaining the spatial distribution information of the agent in the task scenario, dividing it into a discrete grid of N×N, and building a feature matrix and an observation matrix based on global attribute information and local attribute information, weaving it to obtain a state space for reinforcement learning .
It realizes the capture of commonality and individual differences in task scenarios in a multi-agent environment, improves the generalization ability and adaptability of reinforcement learning models, and enables them to be widely used and migrated in multiple task scenarios.
Smart Images
Figure CN119397489A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a generalizable state representation method for game-adversarial intelligent algorithms. Background Art
[0002] In recent years, with the improvement of computing power and the advancement of machine learning technology, the research on general decision frameworks has made significant progress. Researchers have tried to build a general decision framework that can handle a wide range of tasks through various approaches, such as meta-learning, transfer learning, and multi-task learning. Although the research on general decision frameworks has made some progress in the field of reinforcement learning, there are still some significant limitations in practical applications, especially in terms of the generalization ability of the state space.
[0003] The performance of existing general decision-making frameworks in dealing with state space is significantly affected by changes in the number of agents. In a multi-agent environment, changes in the number of agents will lead to drastic changes in the dimension and structure of the state space, which poses a great challenge to the generalization ability of the decision model. Summary of the invention
[0004] The disclosed embodiments provide a generalizable state representation method for a game-playing adversarial intelligent algorithm.
[0005] The first aspect of the embodiments of the present disclosure provides a generalizable state representation method for a game-based confrontation intelligent algorithm, including: Acquire a task scenario, wherein the task scenario includes at least two agents, and includes global attribute information of each of the agents and spatial distribution information of each of the agents; According to the spatial distribution information of each of the intelligent agents, each of the intelligent agents is divided into N×N discrete grids, wherein one of the intelligent agents belongs to one of the discrete grids; According to the global attribute information of the intelligent agent included in each of the discrete grids, an N×N feature matrix corresponding to each of the global attribute information is determined; in the feature matrix corresponding to any of the global attribute information, the value of the element in the i-th row and j-th column is the number of intelligent agents with the global attribute information included in the i-th row and j-th column of the discrete grid; The following processing is performed on each of the agents respectively: according to the discrete grid where the agent is located and the observation range of the agent, an N×N observation matrix corresponding to each local attribute information of the agent is determined; in the observation matrix corresponding to any local attribute information, the value of the element in the i-th row and j-th column is the number of agents in the i-th row and j-th column of the discrete grid that are located within the observation range of the agent and have the local attribute information; A first eigenvector is determined based on each of the feature matrices, and a second eigenvector is determined based on each of the observation matrices. A state space corresponding to the task scenario is obtained based on the fusion of the first eigenvector and the second eigenvector. The state space is used in the reinforcement learning process of the agent.
[0006] The embodiments of the present disclosure have the following advantages: According to the spatial distribution information of each of the agents in the task scene, discretization processing is performed to obtain the discrete grid to which each agent belongs. According to the agents distributed in the discrete grid, the N×N feature matrix corresponding to each of the global attribute information is determined, so as to obtain the global common characteristics of the task scene; according to the agents distributed in the discrete grid and the observation range of the agents, the N×N observation matrix corresponding to each local attribute information of the agent is determined, so as to obtain the local heterogeneous characteristics of each agent in the task scene; by fusing the global common characteristics and the local heterogeneous characteristics, the state space finally used for reinforcement learning is obtained, so that the reinforcement learning model framework relies on the state space, which can not only capture the commonality of the task scene but also adapt to the individual differences of the agents, so that the reinforcement learning model framework can be widely used and migrated in various task scenes, laying a foundation for improving the adaptability and efficiency of the reinforcement learning model in multiple tasks and multiple environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 A schematic flow chart of a generalizable state representation method for a game-adversarial intelligent algorithm provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of a discretization of a task scenario provided in an embodiment of the present disclosure; Figure 3 A schematic diagram of a global common state representation provided in an embodiment of the present disclosure; Figure 4 A schematic diagram showing a local heterogeneous state provided in an embodiment of the present disclosure; Figure 5 A schematic diagram of a general feature vector extraction process for an intelligent agent provided in an embodiment of the present disclosure; Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0008] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.
[0009] As used in this disclosure, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0010] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure.As used in the present disclosure, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
[0011] When the terms “comprising” and / or “made of…” are used in the present disclosure, it specifies the existence of the stated features, integers, steps, operations, elements and / or components, but does not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0012] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as those commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this disclosure.
[0013] The disclosed embodiments provide a generalizable state representation method for game-adversarial intelligent algorithms. The state space is used for reinforcement learning of human intelligence modules, and is particularly suitable for determining the state space used by artificial intelligence models in game-adversarial scenarios.
[0014] like Figure 1 The figure is a flow chart of a generalizable state representation method for a game-adversarial intelligent algorithm provided by an embodiment of the present disclosure. The method mainly includes the following steps: Step 101, obtaining a task scenario, wherein the task scenario includes at least two agents, and includes global attribute information of each of the agents and spatial distribution information of each of the agents.
[0015] Step 102: divide each of the intelligent agents into N×N discrete grids according to the spatial distribution information of each of the intelligent agents, wherein one of the intelligent agents belongs to one of the discrete grids.
[0016] Specifically, each discrete grid corresponds to a spatial region of the same size. According to the spatial position of the agent and the spatial region corresponding to each discrete grid, when the spatial position of the agent falls into the spatial region, the agent is assigned to the discrete grid corresponding to the spatial region.
[0017] In the disclosed embodiment, by dividing the space where all intelligent agents in the task scene are located into N×N discrete grids, a unified and standardized representation of the environmental state is achieved, which can effectively avoid the problem of changes in the representation dimension of the state space caused by changes in the number of intelligent agents or the task environment, and provide stable input features for the artificial intelligence model, thereby improving the generalization ability and adaptability of the artificial intelligence model.
[0018] Wherein, N is an integer greater than 1, and the specific value of N can be pre-configured by the technicians as needed.
[0019] According to the relative position information between the intelligent agent and the discrete grid, if it is determined that the intelligent agent falls into the area where the discrete grid is located, the intelligent agent belongs to the discrete grid.
[0020] A discrete grid may include one or more intelligent agents, or may not include any intelligent agent.
[0021] The spatial distribution information of each intelligent agent may be the relative position information of each intelligent agent or the absolute position information of each intelligent agent. It is only necessary to determine the overall area including each intelligent agent and the position of each intelligent agent in the overall area through the spatial distribution information.
[0022] In an exemplary embodiment, Figure 2 The figure shows a schematic diagram of the discretization of the mission scene, which divides the space where each UAV in the mission scene is located into a 3×3 discrete grid.
[0023] Step 103, based on the global attribute information of the intelligent agent included in each of the discrete grids, determine the N×N feature matrix corresponding to each type of the global attribute information; in the feature matrix corresponding to any of the global attribute information, the value of the element in the i-th row and j-th column is the number of intelligent agents with the global attribute information contained in the i-th row and j-th column of the discrete grid.
[0024] In some embodiments, the global attribute information of the agent includes at least one of the following: the type of the agent; the team information to which the agent belongs; the ability of the agent; the attack method of the agent; the defense method of the agent; and the mounting information of the agent.
[0025] In an exemplary embodiment, in a game confrontation scenario, the team information to which the agent belongs includes the identifier of one of the teams participating in the game confrontation. For example, the red team and the blue team are two teams in the game confrontation. The attribute information of some agents in the task scenario includes the red team identifier, and the attribute information of some agents includes the blue team identifier.
[0026] In an exemplary embodiment, at least one type of intelligent agent, for example, the types of drones include reconnaissance drones, attack drones, defense drones, etc.
[0027] The capabilities of an intelligent agent include at least one capability, for example, the capabilities of a drone include attack capability, reconnaissance capability, defense capability, etc.
[0028] The mounting information of the intelligent body includes the status of the mounted equipment of the intelligent body. For example, the mounting information of a drone includes the mounted machine gun, camera, radar, etc.
[0029] In an exemplary embodiment, the agents are grouped according to their global attribute information, and a corresponding N×N feature matrix is constructed for each group of agents, which is expressed as , the value of the matrix element in the i-th row and j-th column of the feature matrix is: the number of agents belonging to the group in the discrete grid of the i-th row and j-th column. This method realizes the unified and standardized representation of the environmental state, effectively avoids the problem of changes in the state representation dimension caused by changes in the number of agents or changes in the task environment, and provides a basis for improving the generalization ability and adaptability of artificial intelligence models.
[0030] In the embodiment of the present invention, the global common state is represented by an N×N feature matrix corresponding to all global attribute information, which can avoid the impact of changes in task scenarios (such as changes in the number of intelligent agents) on the input feature dimensions of the artificial intelligence model, support diversified task training, have a unified representation and processing of multiple situations, and can cope with input features under various situations, thereby improving the wide adaptability of the artificial intelligence model.
[0031] In the exemplary embodiment, taking the drone attack and defense simulation scenario as an example, Figure 3The figure shows a schematic diagram of the global common state representation. It is assumed that the drone has three types of global attribute information, including type information, capability information and mounting information. Among them, the type information represents different types of drones, such as attack drones, reconnaissance drones and other N types. Each feature matrix corresponding to the type information represents a type, and each matrix element in the feature matrix represents the number statistics of drones of this type in the corresponding discrete grid. It should be noted that the N types here do not mean that they must be the same as the dimension N of the discrete grid. This is just an example, and the number of types is not necessarily related to the dimension N of the discrete grid. Capability information represents the capability characteristics of the drone, such as attack capability, reconnaissance capability, defense capability, etc. The matrix elements in the feature matrix corresponding to each capability information represent the number of drones with this capability. Mounting information represents the mounting status of the drone, such as machine guns, cameras, radars, etc. Similarly, the matrix elements in the feature matrix corresponding to each mounting information represent the number of drones with this mounting information.
[0032] It should be noted that Figure 3 The element values in each feature matrix in are only examples, and the element values in the feature matrices corresponding to different global attribute information may be the same or may be different.
[0033] Step 104, perform the following processing on each of the agents respectively: determine the N×N observation matrix corresponding to each local attribute information of the agent according to the discrete grid where the agent is located and the observation range of the agent; in the observation matrix corresponding to any local attribute information, the value of the element in the i-th row and j-th column is the number of agents in the i-th row and j-th column of the discrete grid that are located within the observation range of the agent and have the local attribute information.
[0034] In some embodiments, the local attribute information of the agent includes at least one of the following: attribute information of a friendly agent of the agent; attribute information of an opponent agent of the agent; and attribute information of the agent itself.
[0035] In an exemplary embodiment, the local attribute information of the agent may be a part or all of the global attribute information.
[0036] In an exemplary embodiment, the attribute information of the intelligent agent itself includes location information and health status information. These two attribute information can accurately describe the state of the intelligent agent in a specific environment and provide support for the artificial intelligence model to formulate more personalized and effective decision-making strategies.
[0037] In the embodiment of the present invention, the observation matrices corresponding to the multiple local attribute information of the intelligent agent are used as the local heterogeneous state representation of the intelligent agent, and the original observation features of the intelligent agent (the original observed local attribute information) are converted into a multi-layer observation matrix, which can capture the local environment and heterogeneity of the intelligent agent in detail.
[0038] In the exemplary embodiment, a multi-layer observation matrix is constructed based on the discretization of the task scene. Describe the local heterogeneous state of the agent. Assume that the original observation characteristics of the agent are expressed as: ,in, Represents the attribute information of the other agent within the observation range of the agent, Represents the attribute information of friendly agents within the observation range of the agent, Represents the attribute information of the agent itself. In different adversarial tasks, and The length of will often vary with task characteristics (e.g., the number of agents), but will not change, therefore, the heterogeneous nature of the agent is reflected in In. Convert the original observation features into a multi-layer observation matrix.
[0039] Specifically, assuming Represents the set of index numbers of all discrete grids, represents a set of agents in a discrete grid. If an agent On a discrete grid Inside, there is .use represents the attribute set of each agent, Represents an agent Properties For any selected attribute Construct the corresponding observation matrix , observe that the value of each matrix element in the matrix is determined by the formula: For example, let In this way, the observation matrix of position features and the observation matrix of health status features are constructed.
[0040] In the exemplary embodiment, taking the drone attack and defense simulation scenario as an example, Figure 4 The figure shows a schematic diagram of the local heterogeneous state representation. Assume that the mission scenario includes three red drones and three blue drones. Assume that the space where these drones are located is divided into 9 discrete grids, and the visible range of each drone is only the adjacent discrete grids. The only two attributes included are location and health status. Figure 4The numbers 1-9 represent the numbers of the discrete grids. The red drones are located in the discrete grids 2 and 3, and the blue drones are located in the discrete grids 5, 6, and 8. The red drone located in the discrete grid 5 is drone No. 1. Each element in the position vector represents the number of the location of the intelligent agent observed by drone No. 1. The blue drone No. 4 located in the discrete grid 3 is not within the observation range of drone No. 1. The position vector of the friendly intelligent agent observed by drone No. 1 is , the observed position vector of the other agent is , the position vector of UAV No. 1 itself is Similarly, each element of the health state vector represents the life value of the agent observed by drone No. 1. Figure 4 In the example, the life value of each drone is 2. The position vector and health state vector are converted into corresponding observation matrices, namely, the position feature matrix and the health state feature matrix.
[0041] Step 105: determine a first eigenvector based on each of the feature matrices, and determine a second eigenvector based on each of the observation matrices, and obtain a state space corresponding to the task scenario based on the fusion of the first eigenvector and the second eigenvector, wherein the state space is used for reinforcement learning of the artificial intelligence model.
[0042] In some embodiments, determining the first eigenvector based on each of the eigenmatrices comprises: Each of the feature matrices is input into a first convolution filter group, and the first feature vector is determined according to an output vector of the first convolution filter group.
[0043] In some embodiments, the first convolution filter group includes P convolution filters, where P is equal to the number of global attribute information of the agent, and N is an integer greater than or equal to 1; The step of inputting each of the feature matrices into the first convolution filter group and determining the first feature vector according to the output vector of the first convolution filter group includes: for each of the feature matrices, selecting a target convolution filter from P convolution filters of the first convolution filter group according to the global attribute information corresponding to the feature matrix, inputting the feature matrix into the target convolution filter, and obtaining a sub-feature vector corresponding to the feature matrix; and summing the sub-feature vectors corresponding to each of the feature matrices to obtain the first feature vector.
[0044] In some embodiments, determining the second eigenvector based on each of the observation matrices comprises: Each of the observation matrices is input into a second convolution filter group, and the second eigenvector is determined according to an output vector of the second convolution filter group.
[0045] In some embodiments, the second convolution filter group includes Q convolution filters, where Q is equal to the number of local attribute information of the agent, and Q is an integer greater than or equal to 1; The step of inputting each of the observation matrices into the second convolution filter group and determining the second eigenvector according to the output vector of the second convolution filter group includes: for each of the observation matrices, selecting a target convolution filter from Q convolution filters of the second convolution filter group according to the local attribute information corresponding to the observation matrix, inputting the observation matrix into the target convolution filter, and obtaining a sub-eigenvector corresponding to the observation matrix; and summing the sub-eigenvectors corresponding to each of the observation matrices to obtain the second eigenvector.
[0046] In the disclosed embodiment, a dual-channel convolution is formed by a first convolution filter group and a second convolution filter group, the feature matrix and the observation matrix are converted into corresponding feature vectors, and the converted feature vectors are fused to obtain a general state space of the intelligent agent.
[0047] Specifically, combined Figure 5 The schematic diagram of the agent general feature vector extraction process shown in the figure explains the dual-channel convolution process as follows: The two-channel convolution consists of the first convolution filter group and the second convolution filter group; the first convolution filter group is expressed as , the second convolution filter bank is expressed as .
[0048] The first convolution filter bank consists of convolution filters Composition, that is ,in is the number of global attribute information, and the calculation formula of the convolution filter is expressed as: , responsible for transforming the feature matrix Convert to the first eigenvector .
[0049] The second convolution filter bank consists of convolution filters, namely , the calculation formula of the convolution filter is expressed as: , which is responsible for transforming the agent's observation matrix Convert to the second eigenvector .
[0050] Finally, the first eigenvector and the second eigenvector are connected to obtain the fused general eigenvector of the agent, that is, the state space ,Right now .
[0051] It should be noted that Figure 5The element values in the observation matrix corresponding to the local heterogeneous state and the feature matrix corresponding to the global common state are only examples, which does not mean that the two matrices must be the same. The corresponding element values in the observation matrix and the feature matrix may be the same or different, depending on the actual statistical situation.
[0052] In the embodiments of the present disclosure, by constructing a universal feature vector that integrates global common state representation and local heterogeneous state representation as the state space, the artificial intelligence model can capture the commonalities between tasks and adapt to individual differences and environmental changes when performing reinforcement learning. It can be effectively migrated and applied in different tasks and environments, so as to promote the development of artificial intelligence models in a wider range of application fields and more diverse fields, overcome the limitations of state space generalization when migrating different task scenarios in existing related technologies, and provide a new solution for reinforcement learning in complex environments.
[0053] The method provided by the embodiment of the present disclosure can not only adapt to changes in the number of intelligent agents, but also effectively deal with the generalization problem of the state space caused by changes in different task types and intelligent agent types, realize efficient formulation and decision-making of the state space of complex environments, and significantly improve the adaptability and efficiency of artificial intelligence models in multiple tasks and multiple environments.
[0054] The step division of the above various methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this disclosure. Adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of this disclosure.
[0055] The present disclosure provides a generalizable state representation device for game-based intelligent algorithms. The specific implementation of the device can be found in the relevant description of the method embodiment, which will not be repeated here. The device mainly includes: An acquisition module, used for acquiring a task scenario, wherein the task scenario includes at least two agents, and includes global attribute information of each of the agents and spatial distribution information of each of the agents; A discretization module, used for dividing each of the intelligent agents into N×N discrete grids according to the spatial distribution information of each of the intelligent agents, wherein one of the intelligent agents belongs to one of the discrete grids; A global processing module, used to determine, according to the global attribute information of the intelligent agent included in each discrete grid, an N×N feature matrix corresponding to each type of the global attribute information; in the feature matrix corresponding to any of the global attribute information, the value of the element in the i-th row and j-th column is the number of intelligent agents with the global attribute information included in the i-th row and j-th column of the discrete grid; The local processing module is used to perform the following processing on each of the agents respectively: determine the N×N observation matrix corresponding to each local attribute information of the agent according to the discrete grid where the agent is located and the observation range of the agent; in the observation matrix corresponding to any local attribute information, the value of the element in the i-th row and j-th column is the number of agents in the i-th row and j-th column of the discrete grid that are located within the observation range of the agent and have the local attribute information; A fusion module is used to determine a first eigenvector based on each of the feature matrices and a second eigenvector based on each of the observation matrices, and to obtain a state space corresponding to the task scenario based on the fusion of the first eigenvector and the second eigenvector, wherein the state space is used in the reinforcement learning process of the agent.
[0056] The functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the method embodiments. The specific implementation and technical effects thereof can be referred to the description of the method embodiments above, and will not be described again here for the sake of brevity.
[0057] It should be noted that all modules involved in this embodiment are logic modules. In practical applications, a logic unit may be a physical unit, or a part of a physical unit, or may be implemented as a combination of multiple physical units. In addition, in order to highlight the innovative part of the present disclosure, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by the present disclosure, but this does not mean that there are no other units in this embodiment.
[0058] Reference Figure 6 , an embodiment of the present disclosure provides an electronic device, comprising: at least one processor 1001; A memory 1002 having at least one program stored thereon, and when the at least one program is executed by the at least one processor, the at least one processor implements the above method; At least one I / O interface 1003 is connected between the processor and the memory and is configured to implement information exchange between the processor and the memory.
[0059] Among them, the processor 1001 is a device with data processing capabilities, including but not limited to a central processing unit (CPU), etc.; the memory 1002 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 1003 is connected between the processor 1001 and the memory 1002, and can realize information interaction between the processor 1001 and the memory 1002, including but not limited to a data bus (Bus), etc.
[0060] In some embodiments, the processor 1001 , the memory 1002 , and the I / O interface 1003 are connected to each other through a bus, and further connected to other components of the computing device.
[0061] This embodiment further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in this embodiment is implemented. To avoid repeated description, the specific steps of the method are not repeated here.
[0062] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the above-described methods may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0063] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0064] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present embodiment and to form different embodiments.
[0065] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.
Claims
1. A generalizable state representation method for game-based intelligent algorithms, characterized in that: include: Acquire a task scenario, wherein the task scenario includes at least two agents, and includes global attribute information of each of the agents and spatial distribution information of each of the agents; According to the spatial distribution information of each of the intelligent agents, each of the intelligent agents is divided into N×N discrete grids, wherein one of the intelligent agents belongs to one of the discrete grids; According to the global attribute information of the intelligent agent included in each of the discrete grids, an N×N feature matrix corresponding to each of the global attribute information is determined; in the feature matrix corresponding to any of the global attribute information, the value of the element in the i-th row and j-th column is the number of intelligent agents with the global attribute information included in the i-th row and j-th column of the discrete grid; The following processing is performed on each of the agents respectively: according to the discrete grid where the agent is located and the observation range of the agent, an N×N observation matrix corresponding to each local attribute information of the agent is determined; in the observation matrix corresponding to any local attribute information, the value of the element in the i-th row and j-th column is the number of agents in the i-th row and j-th column of the discrete grid that are located within the observation range of the agent and have the local attribute information; A first eigenvector is determined based on each of the feature matrices, and a second eigenvector is determined based on each of the observation matrices. A state space corresponding to the task scenario is obtained based on the fusion of the first eigenvector and the second eigenvector, and the state space is used for reinforcement learning of the artificial intelligence model.
2. The method according to claim 1, characterized in that: The global attribute information of the agent includes at least one of the following: the type of the agent; the team information to which the agent belongs; the ability of the agent; the attack method of the agent; the defense method of the agent; and the mounting information of the agent.
3. The method according to claim 1, characterized in that The local attribute information of the agent includes at least one of the following: attribute information of a friendly agent of the agent; attribute information of an opponent agent of the agent; and attribute information of the agent itself.
4. The method according to claim 1, characterized in that The determining of the first eigenvector based on each of the eigenmatrices comprises: Each of the feature matrices is input into a first convolution filter group, and the first feature vector is determined according to an output vector of the first convolution filter group.
5. The method according to claim 4, characterized in that The first convolution filter group includes P convolution filters, where P is equal to the number of global attribute information of the agent, and N is an integer greater than or equal to 1; Inputting each of the feature matrices into a first convolution filter group, and determining the first feature vector according to an output vector of the first convolution filter group, comprises: For each of the feature matrices, according to the global attribute information corresponding to the feature matrix, a target convolution filter is selected from the P convolution filters of the first convolution filter group, and the feature matrix is input into the target convolution filter to obtain a sub-feature vector corresponding to the feature matrix; The sub-eigenvectors corresponding to each of the characteristic matrices are summed to obtain the first eigenvector.
6. The method according to claim 1, characterized in that The determining of the second eigenvector based on each of the observation matrices comprises: Each of the observation matrices is input into a second convolution filter group, and the second eigenvector is determined according to an output vector of the second convolution filter group.
7. The method according to claim 6, characterized in that The second convolution filter group includes Q convolution filters, where Q is equal to the number of local attribute information of the agent, and Q is an integer greater than or equal to 1; Inputting each of the observation matrices into a second convolution filter group, and determining the second eigenvector according to an output vector of the second convolution filter group, comprises: For each of the observation matrices, selecting a target convolution filter from the Q convolution filters of the second convolution filter group according to the local attribute information corresponding to the observation matrix, inputting the observation matrix into the target convolution filter, and obtaining a sub-feature vector corresponding to the observation matrix; The second eigenvector is obtained by summing up the sub-eigenvectors corresponding to each of the observation matrices.