Intelligent reinforcement learning training method based on self-gaming

Abstract

The invention discloses an intelligent reinforcement learning training method based on self-gaming, particularly relates to the field of heterogeneous agent collaborative decision-making, is used for solving the problems of real-time performance and strategy stability in a high-dimensional heterogeneous action space, and reduces the scale of network parameters through typed splitting and shared feature extraction of the action space. Due to the application of the dynamic binary masks, illegal actions are filtered in real time, the hysteresis and exploration deviation depending on a reward punishment mechanism are overcome, and the compliance and high efficiency of the decision making process are ensured; a priority score matrix constructed by fusing tactical stability and environmental complexity is combined with layered importance sampling, so that an action selection strategy is optimized, and the decision stability and environmental adaptability are improved; through introduction of the cross attention network and the collaborative gain coefficient, tactical association between agents is effectively captured and utilized, and the overall efficiency and robustness of collaborative decision making are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of collaborative decision-making of heterogeneous intelligent agents, and more specifically, to an intelligent reinforcement learning training method based on self-game. Background Art

[0002] In heterogeneous agent collaborative decision-making systems, since different types of agents need to perform high-dimensional heterogeneous actions (such as mobile control, resource allocation, and tactical response), traditional methods directly use the joint action space as the overall input to the policy network, resulting in the exponential expansion of the network parameter scale with the number of agents and action dimensions, causing low training efficiency and gradient sparsity problems. When the existing technology adopts an end-to-end joint decision-making framework, it does not decouple the action space of the agent type differences, making it difficult for the policy network to capture the tactical behavior characteristics of a specific type of agent; at the same time, illegal action filtering relies on the post-punishment mechanism of the reward function, which has constraint lag and exploration direction deviation problems. When optimizing the collaborative actions of multiple agents, a linear weighted fusion method is used to ignore the nonlinear tactical dependencies between actions, resulting in a high distortion rate of collaborative decision-making. The above defects seriously restrict the real-time and strategy stability of heterogeneous agent collaborative decision-making in complex confrontation scenarios.

[0003] In order to solve the above problems, a technical solution is now provided. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an intelligent reinforcement learning training method based on self-game, which reduces the scale of network parameters through type-based splitting of action space and shared feature extraction; the application of dynamic binary masks realizes real-time filtering of illegal actions, overcomes the lag and exploration bias of relying on reward and punishment mechanism, and ensures the compliance and efficiency of the decision-making process; the priority scoring matrix constructed by integrating tactical stability and environmental complexity is combined with hierarchical importance sampling to optimize the action selection strategy and improve the stability and environmental adaptability of the decision; the introduction of cross-attention network and collaborative gain coefficient effectively captures and utilizes the tactical correlation between intelligent agents, enhances the overall effectiveness and robustness of collaborative decision-making, so as to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: Split the overall action space into several subspaces according to the agent type label, and generate the original action probability matrix of each subspace; In the real-time environment, illegal actions are filtered out to obtain the legal action sequence set of each subspace; In the legal action sequence, actions that take into account both tactical stability and environmental adaptability are prioritized to generate action vectors; The action vector is input into the cross-attention network to calculate the tactical relevance matrix, and the collaborative gain coefficient is generated through matrix multiplication to correct the original action value, and the final collaborative decision instruction set is output.

[0006] In a preferred embodiment, the acquisition of the original action probability matrix includes the following contents: Under the guidance of the agent type label, the overall action space is split into multiple independent subspaces; after the split, the actions of each subspace are no longer coupled with each other, and each subspace corresponds to an independent policy network; for the generation of the action probability matrix in each subspace, the global environment state is first input into the shared convolutional layer for feature extraction to obtain the environment feature representation; then, for each subspace, independent linear mapping weights and bias terms are applied to transform the extracted features, and finally the original action probability matrix of the corresponding subspace is generated.

[0007] In a preferred embodiment, the process of obtaining the legal action sequence set of each subspace is as follows: According to the analysis results of the real-time environment status and the tactical rule library, a dynamic binary mask matrix is ​​generated; if an action meets the current environment and tactical requirements, the mask value of the corresponding action is 1; otherwise, the mask value is 0; the original action probability matrix is ​​multiplied element by element with the dynamic binary mask matrix through the Hadamard product, ensuring that only legal actions are retained and illegal actions are set to zero; then, the Softmax function is applied to renormalize the filtered matrix to ensure that the sum of the probabilities of all legal actions is 1; the action probability matrix after the Hadamard product and renormalization is the set of legal action sequences in each subspace; each subspace only retains legal actions under the current environment and tactical rules.

[0008] In a preferred embodiment, the tactical stability analysis is performed by calculating the tactical stability index, and the environmental adaptability analysis is performed by calculating the complexity sensitivity index.

[0009] In a preferred embodiment, the logic for obtaining the tactical stability index is as follows: The historical action sequence of the agent is divided into the first half and the second half in chronological order; for these two subsequences, the dynamic time warping algorithm is applied to find the optimal alignment path between the two sequences and calculate the cumulative distance along this path; the cumulative distance is obtained by comparing the differences of corresponding points in the sequence and accumulating them point by point; the action sequence is converted into the state transition form of continuous action pairs, the number of occurrences of each action pair is counted, and the frequency of occurrence of each action pair is calculated; based on the frequency distribution, the Shannon entropy formula is applied to calculate the entropy value of the state transition; the inverse of the dynamic time warping distance is combined with the negative exponent of the Shannon entropy value to calculate the tactical stability index.

[0010] In a preferred embodiment, the calculation process of the complexity sensitivity index is as follows: For each environmental complexity indicator, the logistic mapping formula is applied for multiple iterations to generate a chaotic sequence; the indicators after chaotic transformation are regarded as nodes of the graph, and an adjacency matrix is ​​constructed, and the matrix elements are the inverses of the distances between nodes; based on the adjacency matrix, the Laplace matrix of the graph is calculated, specifically the degree matrix minus the adjacency matrix; the eigenvalues ​​of the Laplace matrix are solved, and the second smallest eigenvalue, called algebraic connectivity, is extracted; the algebraic connectivity is ratioed with the sum of the indicators after chaotic transformation to calculate the complexity sensitivity index.

[0011] In a preferred embodiment, the process of obtaining the motion vector is as follows: Combine the tactical stability index with the complexity sensitivity index to calculate the priority score of each agent type; form a vector of the priority scores of all agent types, apply the Softmax function to normalize them, and use the normalized result as the priority score matrix; multiply the original probability by the score of the corresponding type in the priority score matrix element by element to obtain the adjusted probability distribution; normalize the adjusted distribution, generate random numbers and sample specific actions to form an action vector.

[0012] In a preferred embodiment, the action vector is input into a cross-attention network to calculate the tactical relevance between agents. The specific process is as follows: for two agents, their action vectors are converted into query vectors, key vectors and value vectors respectively; the attention weight is obtained by calculating the dot product of the query vector and the key vector; the attention weight is normalized and weightedly summed with the value vector to generate a tactical relevance vector for a single agent; the tactical relevance vectors of all agents are integrated to form a tactical relevance matrix, in which the elements represent the tactical relevance strength between agents.

[0013] In a preferred embodiment, based on the tactical correlation matrix, a synergy gain coefficient is generated by matrix multiplication operation to correct the action value. The specific steps are: multiplying the tactical correlation matrix with the action vector matrix to generate a synergy correction vector, the elements of which represent the action adjustment value of the intelligent agent after considering the tactical correlation; then, weighted fusion of the synergy correction vector and the original action value is performed to generate a synergy gain coefficient.

[0014] In a preferred embodiment, the collaborative gain coefficient is input into the decision network to generate a final collaborative decision instruction set; the decision network converts the collaborative gain coefficient into an action probability distribution, and its internal parameters include a weight matrix and a bias vector for adjusting the characteristics of the probability distribution; based on the generated action probability distribution, a collaborative decision instruction set is formed by selecting the action with the highest probability.

[0015] The technical effects and advantages of the intelligent reinforcement learning training method based on self-game of the present invention are as follows: The present invention significantly improves the performance and efficiency of the heterogeneous agent collaborative decision-making system by combining action space decoupling, dynamic mask filtering, priority fusion and collaborative attention mechanism. First, through the typed splitting of the action space and shared feature extraction, the scale of network parameters is reduced, and the low training efficiency and gradient sparsity problems caused by the high-dimensional joint action space in the traditional method are alleviated. Secondly, the application of dynamic binary mask realizes real-time filtering of illegal actions, overcomes the lag and exploration bias of the reward and punishment mechanism, and ensures the compliance and efficiency of the decision-making process. Furthermore, the priority score matrix constructed by integrating tactical stability and environmental complexity is combined with hierarchical importance sampling to optimize the action selection strategy and improve the stability and environmental adaptability of the decision. Finally, the introduction of cross-attention network and collaborative gain coefficient effectively captures and utilizes the tactical correlation between agents, and enhances the overall effectiveness and robustness of collaborative decision-making. In complex confrontation scenarios, the real-time decision-making ability, strategy stability and collaborative optimization level of the heterogeneous agent system are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the intelligent reinforcement learning training method based on self-game of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Embodiment 1: Figure 1 The present invention provides an intelligent reinforcement learning training method based on self-game, comprising: S1: Split the overall action space into several subspaces according to the agent type label, and generate the original action probability matrix of each subspace.

[0019] S2: In the real-time environment, filter out illegal actions and obtain the set of legal action sequences in each subspace.

[0020] S3: In the legal action sequence, give priority to actions that take into account both tactical stability and environmental adaptability to generate action vectors.

[0021] S4: Input the action vector into the cross-attention network to calculate the tactical correlation matrix, generate the collaborative gain coefficient through matrix multiplication to correct the original action value, and output the final collaborative decision instruction set.

[0022] In complex scenarios of multi-agent collaborative decision-making, different types of agents usually have different task goals, action spaces, and behavioral strategies. Due to the functional differences of these agents, directly merging all action spaces into a unified input will lead to parameter expansion, increasing the computational overhead and gradient sparsity problems during training. Step S1 splits the overall action space according to the agent type label and extracts environmental information in combination with a shared convolutional layer, independently assigning a policy network to each subspace while maintaining the consistency of global environmental features.

[0023] Multi-agent collaborative decision-making refers to a situation in which multiple independent agents make decisions together through cooperation and coordination in a complex environment to achieve the overall goal or optimize system performance. In this scenario, each agent usually has different functions, tasks and behavior spaces, and there is both information exchange and mutual influence between agents. According to the state of the environment and the behavior of other agents, through sharing information, collaborative planning and division of labor, a comprehensive response to the environment is ultimately achieved. Multi-agent collaborative decision-making is widely used in robot swarms, autonomous driving, energy scheduling, financial investment and other fields. The core challenge lies in how to deal with the interactions and conflicts between agents while maintaining the efficiency and stability of the decision-making process.

[0024] Step S1 includes the following contents: Agent types are divided according to their functions and the tasks they perform. Each type of agent has a different range of action and decision space, and the typical division is based on the platform's functions, tasks, and required operational dimensions. For example, if a complex task involves multiple roles such as patrolling, reconnaissance, and command, each role is an agent type with a unique action space related to the task. Each agent type label provides the necessary information for subsequent action space splitting and policy training to ensure that the policy network of each subspace can focus on a specific behavior pattern and avoid unnecessary cross-interference.

[0025] Under the guidance of the agent type label, the overall action space is split into multiple independent subspaces. Each subspace represents a set of actions of a specific type of agent, and all actions in the action set have similar scope of influence and decision logic. After the split, the actions of each subspace are no longer coupled with each other, and each subspace corresponds to an independent policy network. This method effectively avoids the problems of gradient sparsity and low training efficiency caused by high-dimensional joint action space in traditional methods. At the same time, the independence of subspace splitting reduces the interference of action selection of various agents on the strategies of other agents, and improves the stability of the training process.

[0026] In order to make the policy network of each subspace perceive the global environment state uniformly, but not be affected by the differences between different types of agents, the environmental situation features are extracted through a shared convolution layer. The shared convolution layer can transmit consistent environmental information between different subspaces, ensuring that the policy training of all agents is based on the same global environment understanding. For the generation of the action probability matrix in each subspace, the global environment state is first input into the shared convolution layer for feature extraction to obtain the environmental feature representation. Then, for each subspace, independent linear mapping weights and bias terms are applied to transform the extracted features, and finally the original action probability matrix of the corresponding subspace is generated.

[0027] By splitting the overall action space into independent subspaces for different agent types and using shared convolutional layers to extract environmental information, this step effectively reduces the complexity of the policy network during training and improves training efficiency. The policy network in each subspace focuses on the behavioral characteristics of a specific type of agent while maintaining a unified perception of the global environment, thus providing stable and clear input for subsequent tactical rule parsing and priority decision-making.

[0028] Step S2 includes the following contents: In the complex multi-agent collaborative decision-making process, the real-time environment state and the preset tactical rules jointly affect the decision-making of the agent. The goal of step S2 is to generate a legal action set by combining tactical rules with the environment state. The original action probabilities are screened and normalized through a dynamic binary mask matrix to ensure that each agent performs actions that meet the current environment and task requirements.

[0029] The real-time environment status refers to the dynamic characteristics of the current environment, usually including the status information of factors such as position, speed, obstacles, enemy and friendly units. This information is collected and updated in real time through sensors, visual systems or other monitoring methods. The tactical rule base is a database containing various strategy rules. These rules determine the allowed action trajectories and behavior restrictions based on different tactical goals, mission requirements and environmental constraints. Based on these rules, the tactical rule base provides effective behavior guidance for the current situation every time the environment is updated.

[0030] A dynamic binary mask matrix is ​​generated based on the real-time environment status and the results of the analysis of the tactical rule base. Each element in the matrix indicates whether a certain action is legal in a specific environment. When the real-time environment changes, the mask matrix will be updated according to the rules in the tactical rule base and the current environmental conditions (such as the position of the agent, the distribution of the target, etc.). If an action meets the current environment and tactical requirements, the mask value corresponding to the action is 1; otherwise, the mask value is 0. The generation method of the mask matrix can be expressed as: in, It's action is the environmental state, Indicates whether the action is valid in this state.

[0031] The original action probability matrix is ​​multiplied element-wise with the dynamic binary mask matrix through the Hadamard product. The Hadamard product ensures that only legal actions are retained and illegal actions are set to zero. Then, the Softmax function is applied to renormalize the filtered matrix to ensure that the sum of the probabilities of all legal actions is 1. This process is performed by the following formula: in, is the action probability after mask filtering and Softmax renormalization, is the mask matrix, ensuring that the probability of illegal actions is 0.

[0032] The action probability matrix after Hadamard product and renormalization is the set of legal action sequences in each subspace. Each subspace only retains legal actions under the current environment and tactical rules. Through this set of legal actions, the subsequent decision-making process can effectively avoid executing invalid or incompatible actions.

[0033] Step S2 dynamically generates a set of legal action sequences by combining the real-time environment status and the tactical rule base. The mask matrix is ​​used to filter out actions that meet the current environment and task requirements. The Hadamard product and Softmax renormalization operations ensure that only legal actions can be selected in the final output action probability matrix. This process effectively reduces the action space of the agent, improves the real-time and stability of decision-making, and provides high-quality input for subsequent decision-making steps.

[0034] Step S3 includes the following contents: The tactical stability analysis is conducted by calculating the tactical stability index, and the environmental adaptability analysis is conducted by calculating the complexity sensitivity index.

[0035] The logic for obtaining the tactical stability index is as follows: The calculation of the tactical stability index aims to quantify the consistency and predictability of the agent's action selection pattern in historical decision-making, thereby evaluating the stability of tactical behavior. The design combines the morphological analysis of time series and the entropy measurement of information theory to ensure that the evaluation reflects both the continuity of action patterns and the certainty of selection behavior. The calculation process is divided into two stages: analyzing the temporal dynamic stability of action sequences and quantifying the degree of structural chaos in action selection, and finally integrating the two parts to generate a priority score.

[0036] The historical action sequence of the agent is divided into the first half and the second half in chronological order. For these two subsequences, the dynamic time warping algorithm (DTW) is applied to find the optimal alignment path between the two sequences and calculate the cumulative distance along this path. The cumulative distance is obtained by comparing the differences between corresponding points in the sequence and accumulating them point by point. The smaller the distance value, the less the change in the previous and subsequent action patterns, indicating a higher stability in time. The specific processing process is: Given an agent type In the time window Action sequence within ,in , Indicates the total number of action types; Represents a time window The length of the action sequence. Divide the sequence into the first half and the second half . Calculate the dynamic time warping (DTW) distance between two subsequences DTW measures the similarity of action patterns through the optimal alignment path. The smaller the cumulative distance, the more consistent the previous and subsequent patterns are.

[0037] Convert the action sequence into the state transition form of continuous action pairs, count the number of occurrences of each action pair, and calculate the frequency of occurrence of each action pair. Based on the frequency distribution, the Shannon entropy formula is used to calculate the entropy value of the state transition. The formula is the sum of each frequency multiplied by the logarithm of the corresponding frequency and take the negative value. The lower the entropy value, the more orderly the state transition, indicating that the stability of the action selection is higher. Combine the reciprocal of the dynamic time warp distance with the negative exponent of the Shannon entropy value to calculate the tactical stability index. The specific operation is: take the reciprocal of the dynamic time warp distance, multiply it by the exponential form of the Shannon entropy value after taking the natural logarithm and then taking the negative value, and the result is used as the final index. The design ensures that the smaller the distance and the lower the entropy value, the larger the index, reflecting higher tactical stability. The specific processing process is: The action sequence is converted into a state transition probability distribution, where a state is defined as a continuous action pair. . Count all state pairs frequency ,in Indicates state pairs. Calculate the Shannon entropy: , Tactical Stability Index The fusion is done by the following formula: ,in, and are the action indices for the first and second half of the action sequence.

[0038] The calculation process of complexity sensitivity index is as follows: The calculation of the complexity sensitivity index aims to evaluate the impact of environmental complexity on the decision-making of intelligent agents by analyzing the nonlinear dynamic characteristics of environmental features and the coupling relationship between indicators. Chaos theory is used to amplify the unpredictability of the environment, and the structural complexity between indicators is quantified through graph theory to ensure that the evaluation takes into account both the dynamics of a single indicator and the correlation of multiple indicators. The calculation process is divided into two stages: nonlinear transformation of environmental indicators, and analysis of the interaction between the transformed indicators, and finally the fusion results are generated to generate an index.

[0039] For each environmental complexity index, the logistic mapping formula is applied for multiple iterations to generate a chaotic sequence. Logistic mapping sets initial values ​​and parameters, and calculates the next iteration value in sequence, with a fixed number of iterations. The chaotic sequence amplifies small fluctuations in the index, reflects the potential instability of the environment, and generates results as the basis for subsequent analysis. The specific processing process is: Assume that the environmental complexity is determined by the indicator Indicates that each is a scalar value. Applying the logistic map to each indicator generates a chaotic sequence: , initial value (normalized to [0,1]), represents the number of iteration steps, is the parameter of the logistic map, used for chaos map transformation, usually with a value of 3.9; iterative = 10 times, take This transformation amplifies the nonlinear characteristics of the indicator and simulates the dynamic effects of complex environments.

[0040] The indicators after chaotic transformation are regarded as nodes of the graph, and an adjacency matrix is ​​constructed, where the elements of the matrix are the inverses of the distances between nodes. Based on the adjacency matrix, the Laplace matrix of the graph is calculated, which is the degree matrix minus the adjacency matrix. Solve the eigenvalues ​​of the Laplace matrix and extract the second smallest eigenvalue, which is called algebraic connectivity. The larger the eigenvalue, the stronger the coupling between the indicators and the more uniform the complexity distribution. The algebraic connectivity is compared with the sum of the indicators after chaotic transformation to calculate the complexity sensitivity index. The specific operation is: replace the algebraic connectivity divided by the sum of all chaotic sequence values, and the result is used as the final index. When the design ensures that the indicators are tightly coupled and the total complexity is moderate, the index is higher, reflecting the sensitivity of the environment to the decision. The specific processing process is: The transformed index Treat them as graph nodes and construct an adjacency matrix ,in . Calculate the Laplacian matrix ( is the degree matrix), find its second smallest eigenvalue Complexity Sensitivity Index Defined as: ,in Reflects the coupling strength between indicators. The larger the value, the more uniform the complexity distribution. and Represents the index of the node in the graph, each node corresponds to the environmental index after the chaotic map transformation and ; The number of indicators that represent the complexity of the environment; the denominator normalizes the total complexity. The interactions between indicators are captured through graph theory.

[0041] The construction of the priority score matrix aims to integrate the tactical stability index and the complexity sensitivity index to provide a quantitative basis for the selection of the agent type. The interaction between the two indexes is introduced through the nonlinear fusion formula to ensure that the score reaches its peak when the tactical stability is high and the complexity sensitivity is moderate, taking into account both stability and adaptability. The calculation process includes two steps: nonlinear fusion and normalization.

[0042] Combine the tactical stability index with the complexity sensitivity index to calculate the priority score for each agent type. The specific operation is: take the tactical stability index divided by the complexity sensitivity index input into the sigmoid function, where the sigmoid function is 1 divided by (1 plus the negative exponential power of e). This calculation makes the score most sensitive to tactical stability when the complexity sensitivity index is close to a certain threshold. The priority scores of all agent types are formed into a vector and normalized using the Softmax function. The Softmax function calculates the exponential value of each score and divides it by the sum of all score exponential values ​​to generate a probability distribution. The normalized result is used as the priority score matrix to ensure that the sum of the scores is 1 for subsequent operations. The specific processing process is: Tactical Stability Index and complexity sensitivity index Fusion, calculation of priority score , using the nonlinear formula: When the complexity sensitivity index is high, the denominator increases, reducing the priority score; when the tactical stability index is high, the priority score increases. The Sigmoid function introduces nonlinear interaction to enhance the dynamic balance of the two indexes. For all agent types , construct the score vector , normalized to a probability distribution through Softmax: .

[0043] According to the probability distribution in the priority score matrix, the roulette wheel method is applied to select the agent type. The roulette wheel method generates a random number between 0 and 1, and selects the corresponding type based on the probability accumulation and determines which interval it falls into. In the set of legal actions of the selected type, the original action probability distribution is obtained. The original probability is multiplied element by element by the score of the corresponding type in the priority score matrix to obtain the adjusted probability distribution. The adjusted distribution is normalized to ensure that the sum of the probabilities is 1. Based on the normalized distribution, random numbers are generated and specific actions are sampled to form an action vector. The specific processing process is: According to the probability distribution Sampling Agent Types ,exist In the set of legal actions, according to the adjusted probability distribution Sampling Action , construct the action vector.

[0044] Step S3 generates a tactical stability index by analyzing the behavioral trajectory of the legal action sequence, and combines it with the complexity sensitivity index obtained by analyzing the real-time sensor data to construct a priority score matrix to provide quantitative guidance for action selection. At the same time, the layered importance sampling technology is used to optimize the generation of action vectors to ensure that the decision-making takes into account both stability and adaptability, thereby effectively making up for the shortcomings of traditional methods in high-dimensional heterogeneous action space and improving the real-time and strategy stability of the heterogeneous intelligent collaborative decision-making system in complex confrontation scenarios.

[0045] In the collaborative decision-making system of heterogeneous agents, the generation of action vectors is an important part of the decision-making process. In step S3, a priority score matrix is ​​constructed by integrating the tactical stability index and the complexity sensitivity index, and an action vector is generated using the hierarchical importance sampling technique. However, the action vector only reflects the individual decision-making tendency of the agent in the current environment, and does not fully consider the synergy and tactical relevance between agents. Step S4 aims to capture the tactical relevance between agents by introducing a cross-attention network, generate a synergy gain coefficient to correct the original action value, and finally output a collaborative decision instruction set, thereby improving the synergy and overall effectiveness of decision-making, and ensuring stability and real-time performance in complex confrontation scenarios.

[0046] Step S4 includes the following contents: The action vector generated in the previous step is fed into the cross-attention network to calculate the tactical relevance between agents. The cross-attention network operates through a specific mechanism. The specific process is as follows: for two agents, their action vectors are converted into query vectors, key vectors, and value vectors respectively. The attention weight is obtained by calculating the dot product of the query vector and the key vector. This weight reflects the degree of tactical attention of one agent to another. The attention weight is normalized and weighted summed with the value vector to generate the tactical relevance vector of a single agent. The tactical relevance vectors of all agents are integrated to form a tactical relevance matrix, in which the elements represent the strength of tactical relevance between agents. This process uses the attention mechanism to accurately capture the dynamic tactical dependencies between agents.

[0047] Based on the tactical correlation matrix, a synergy gain coefficient is generated through matrix multiplication to correct the action value. The specific steps are: multiply the tactical correlation matrix with the action vector matrix to generate a synergy correction vector, in which the elements represent the action adjustment value of the agent after considering the tactical correlation. Subsequently, the synergy correction vector is weightedly fused with the original action value to generate a synergy gain coefficient. During the fusion process, the degree of influence of the synergy effect is controlled by adjusting the parameters to ensure that the corrected action value reflects both individual decision-making and group synergy effects. Weighted fusion uses a linear combination method to reflect the gain adjustment effect of tactical correlation on the action value.

[0048] The collaborative gain coefficient is input into the decision network to generate the final collaborative decision instruction set. The decision network converts the collaborative gain coefficient into an action probability distribution. Its internal parameters include the weight matrix and the bias vector, which are used to adjust the characteristics of the probability distribution. According to the generated action probability distribution, the collaborative decision instruction set is formed by sampling or selecting the action with the highest probability. This process ensures that the instruction set reflects both the optimal decision at the individual level and the result of group collaborative optimization.

[0049] The collaborative decision-making instruction set is generated through iterative optimization in a multi-agent system to coordinate the behavior of agents and improve decision-making efficiency. The method of use is to represent the final instruction set as a set of multiple scalar values, each of which is input into the decision module of the agent as a control parameter to adjust the probability distribution of action selection. Iterative optimization is achieved through weight function and standardization. The weight function calculates the next generation value based on the result of the previous iteration and the normal distribution parameter, and the standardization ensures that the scalar value range is between 0 and 1. After multiple iterations, the instruction set converges to a stable state, guiding the group of agents to reach the global optimal decision in a complex dynamic environment. Its role is to enhance the coordination and adaptability of decision-making behavior and reduce conflicts and resource waste. In high-dimensional heterogeneous action space, the application of instruction sets significantly improves the overall efficiency and real-time performance of group decision-making. It is suitable for distributed decision-making systems, especially in complex confrontation scenarios that require efficient collaboration, to ensure that the system responds quickly and operates stably, and ultimately achieves optimized decision-making efficiency and global optimal state.

[0050] Step S4 calculates the tactical correlation matrix through the cross-attention network, captures the tactical correlation between agents, generates the collaborative gain coefficient by matrix multiplication to correct the original action value, and finally outputs the collaborative decision instruction set. This method effectively integrates individual decision-making and group synergy, makes up for the shortcomings of traditional methods in action collaborative optimization, improves the decision-making stability and real-time performance of heterogeneous agent systems in complex environments, and provides technical support for collaborative decision-making.

[0051] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0052] It should be noted that the system of the present invention can be deployed on the device itself to realize embedded applications, and can also be run on a PC or other terminal with a user interface, thereby meeting a variety of hardware environments and usage requirements.

[0053] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0054] It should be noted that, in this article, if there are relational terms such as first and second, etc., they are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0055] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An intelligent reinforcement learning training method based on self-game, characterized in that: Includes steps: Split the overall action space into several subspaces according to the agent type label, and generate the original action probability matrix of each subspace; In the real-time environment, illegal actions are filtered out to obtain the legal action sequence set of each subspace; In the legal action sequence, actions that take into account both tactical stability and environmental adaptability are prioritized to generate action vectors; The action vector is input into the cross-attention network to calculate the tactical relevance matrix, and the collaborative gain coefficient is generated through matrix multiplication to correct the original action value, and the final collaborative decision instruction set is output.

2. The intelligent reinforcement learning training method based on self-game according to claim 1 is characterized in that: The acquisition of the original action probability matrix includes the following: Under the guidance of agent type labels, the overall action space is split into multiple independent subspaces; After the split, the actions of each subspace are no longer coupled to each other, and each subspace corresponds to an independent policy network. For the generation of the action probability matrix in each subspace, the global environment state is first input into the shared convolutional layer for feature extraction to obtain the environment feature representation. Then, for each subspace, independent linear mapping weights and bias terms are applied to transform the extracted features, and finally the original action probability matrix of the corresponding subspace is generated.

3. The self-game-based intelligent reinforcement learning training method according to claim 2, characterized in that: The process of obtaining the set of legal action sequences in each subspace is as follows: Generate a dynamic binary mask matrix based on the real-time environment status and the analysis results of the tactical rule library; if an action meets the current environment and tactical requirements, the mask value of the corresponding action is 1; Otherwise, the mask value is 0; the original action probability matrix is ​​multiplied element by element with the dynamic binary mask matrix through the Hadamard product to ensure that only legal actions are retained and illegal actions are set to zero; then, the Softmax function is applied to renormalize the filtered matrix to ensure that the sum of the probabilities of all legal actions is 1; the action probability matrix after the Hadamard product and renormalization is the set of legal action sequences in each subspace; each subspace only retains legal actions under the current environment and tactical rules.

4. The intelligent reinforcement learning training method based on self-game according to claim 3 is characterized in that: The tactical stability analysis is conducted by calculating the tactical stability index, and the environmental adaptability analysis is conducted by calculating the complexity sensitivity index.

5. The self-game-based intelligent reinforcement learning training method according to claim 4, characterized in that: The logic for obtaining the tactical stability index is as follows: The historical action sequence of the agent is divided into the first half and the second half in chronological order; for these two subsequences, the dynamic time warping algorithm is applied to find the optimal alignment path between the two sequences and calculate the cumulative distance along this path; the cumulative distance is obtained by comparing the differences of corresponding points in the sequence and accumulating them point by point; the action sequence is converted into the state transition form of continuous action pairs, the number of occurrences of each action pair is counted, and the frequency of occurrence of each action pair is calculated; based on the frequency distribution, the entropy value of the state transition is calculated by applying the Shannon entropy formula; The tactical stability index is calculated by combining the inverse of the dynamic time warp distance with the negative exponential of the Shannon entropy value.

6. The self-game-based intelligent reinforcement learning training method according to claim 5, characterized in that: The calculation process of complexity sensitivity index is as follows: For each environmental complexity index, the logistic mapping formula is applied for multiple iterations to generate a chaotic sequence. The index after chaotic transformation is regarded as a node of the graph, and an adjacency matrix is ​​constructed. The matrix elements are the inverse of the distance between nodes. Based on the adjacency matrix, the Laplacian matrix of the graph is calculated, which is the degree matrix minus the adjacency matrix; the eigenvalues ​​of the Laplacian matrix are solved, and the second smallest eigenvalue is extracted, which is called algebraic connectivity; the algebraic connectivity is ratioed with the sum of the indicators after the chaotic transformation to calculate the complexity sensitivity index.

7. The self-game-based intelligent reinforcement learning training method according to claim 6, characterized in that: The process of obtaining the action vector is as follows: Combine the tactical stability index with the complexity sensitivity index to calculate the priority score of each agent type; form a vector of the priority scores of all agent types, apply the Softmax function to normalize them, and use the normalized result as the priority score matrix; multiply the original probability by the score of the corresponding type in the priority score matrix element by element to obtain the adjusted probability distribution; normalize the adjusted distribution, generate random numbers and sample specific actions to form an action vector.

8. The self-game-based intelligent reinforcement learning training method according to claim 7, characterized in that: The action vectors are fed into the cross-attention network to calculate the tactical relevance between the agents. The specific process is as follows: for the two agents, their action vectors are converted into query vectors, key vectors, and value vectors respectively; the attention weight is obtained by calculating the dot product of the query vector and the key vector; After normalization, the attention weights are weighted and summed with the value vector to generate the tactical relevance vector of a single agent. The tactical relevance vectors of all agents are integrated to form a tactical relevance matrix, in which the elements represent the strength of the tactical relevance between agents.

9. The self-game-based intelligent reinforcement learning training method according to claim 8, characterized in that: Based on the tactical correlation matrix, a synergy gain coefficient is generated through matrix multiplication operation to correct the action value. The specific steps are: multiply the tactical correlation matrix with the action vector matrix to generate a synergy correction vector, in which the elements represent the action adjustment value of the intelligent agent after considering the tactical correlation; then, the synergy correction vector is weightedly fused with the original action value to generate a synergy gain coefficient.

10. The self-game-based intelligent reinforcement learning training method according to claim 9, characterized in that: Input the collaborative gain coefficient into the decision network to generate the final collaborative decision instruction set; The decision network converts the collaborative gain coefficient into an action probability distribution. Its internal parameters include a weight matrix and a bias vector, which are used to adjust the characteristics of the probability distribution. According to the generated action probability distribution, a collaborative decision instruction set is formed by selecting the action with the highest probability.

Images