Distributed joint combat deduction engine architecture based on multi-agent reinforcement learning

Abstract

The invention relates to the technical field of deduction engines, in particular to a distributed joint combat deduction engine architecture based on multi-agent reinforcement learning, which comprises a data processing layer capable of acquiring target area environment data, entity historical state data and trust data, target area environment data and entity historical state data are processed through a distributed computing framework, and original trust vectors are generated through trust data; the simulation engine layer can perform environment modeling and entity modeling through an entity modeling engine and an environment simulation engine; carrying out credibility dynamic modeling based on the historical behaviors of the intelligent agent; the joint command layer can coordinate and dispatch a target environment entity and environment resources through a coordination engine according to a preset command scheme, and performs cross-domain trust conduction; and the analysis and evaluation layer can deduce and restore in real time, predict the situation of the target area and evaluate the robustness of the command scheme.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deduction engine, in particular to a distributed joint operation deduction engine architecture based on multi-agent reinforcement learning. BACKGROUND

[0002] The joint operation deduction engine is a core technology platform supporting modern military simulation deduction, and the architecture design directly affects the authenticity, real-time performance, scalability and decision support capability of the deduction.

[0003] The trust degree of each agent in the deduction process not only relates to the multi-agent cooperation efficiency problem, but more importantly, it relates to the reliability of the decision. Moreover, when the deduction scene changes dynamically (such as entering urban street fighting) or a sudden event (communication interruption) occurs, the original trust evaluation model may be implemented.

[0004] Therefore, we propose a technical solution to solve the adaptability of trust evaluation and crisis response robustness when the scene migrates by establishing a scene adaptive management mechanism. SUMMARY

[0005] The present application solves the problem of trust degree between multi-agents in the deduction engine through hierarchical design algorithm.

[0006] The technical solution proposed by the present application is: a distributed joint operation deduction engine architecture based on multi-agent reinforcement learning, comprising: A data processing layer, the data processing module is configured to be able to acquire target area environment data, entity historical state data and trust data, process the target area environment data and entity historical state data through a distributed computing framework, and generate an original trust vector using the trust data; A simulation engine layer, the simulation engine layer is configured to be able to perform environment modeling and entity modeling through an entity modeling engine and an environment simulation engine; and dynamically model the trust degree based on the historical behavior of the agent; A joint command layer, the joint command layer is configured to be able to coordinate and schedule target environment entities and environment resources according to a preset command scheme through a coordination engine, and conduct cross-domain trust conduction; An analysis and evaluation layer, the analysis and evaluation layer is configured to be able to real-time deduction review and predict the target area situation, and evaluate the robustness of the command scheme.

[0007] Preferably, the target area environment data, entity historical state data and trust data are acquired, comprising: Acquire low-latency target area environment data and entity historical state data through a data distribution service (DDS); Real-time collection of trust data through DDS middleware, wherein the trust data includes communication response delay, task completion rate and resource occupation rate of the preset agent in the data processing layer; Constructing a trust data set ; wherein, represents the communication response delay, task completion rate and resource occupation rate of the i-th agent.

[0008] Preferably, the processing of the target area environment data and the entity historical state data through the distributed computing framework, and the generation of the original trust vector using the trust data, include: Storing the battlefield environment data, the entity historical state data and the trust data through a database; Processing the target area environment data and the entity historical state data using HPC or cloud computing; Determining the trust data weight through an entropy weight method ; wherein, represents the entropy value of the i-th trust data; represents the communication response delay, task completion rate and resource occupation rate of the i-th agent. represents the normalized value of the i-th trust data of the i-th agent; represents the number of agents; represents the number of elements in the trust data set; Generating the original trust vector: ; wherein, represents the trust vector of the i-th agent.

[0009] Preferably, the environment modeling and entity modeling through the entity modeling engine and the environment simulation engine include: The entity modeling engine includes a physical model and a behavior model, wherein the physical model includes a sensor probability model and a ballistics equation; and the behavior model includes a finite state machine (FSM) and a belief-desire-intention (BDI) decision model; The environment simulation engine includes a dynamic environment model and a geographic information system; Simulating the motion state of the entity through the physical model and the behavior model; Simulating the real-time meteorological state and electromagnetic interference state through the dynamic environment model; Providing ground elevation data through the geographic information system, so as to provide data basis for path planning and line-of-sight calculation.

[0010] Preferably, the dynamic modeling of the trust degree based on the historical behavior of the agent includes: Based on the trust data of the agent, a moment trust decay model is constructed to solve the problem of the credibility decline of the historical trust data.​​​​ Construct a trust decay model, namely: Let the trust level of the current trust data be set. ; in, This represents the number of successful collaborations between intelligent agents. This indicates the number of failed collaborations between intelligent agents. Indicates the attenuation coefficient. Indicates the time interval since the last exchange; Introduce a deceptive behavior monitor to track deviations from abnormal behaviors, including: Deceptive behavior monitor ;in, Indicates the actual value of the behavior. Indicates a reasonable threshold for behavior. Indicates the number of historical collaboration failures; This indicates the number of collaboration failures after the correction. Indicates the degree of deviation from the behavior, when At that time, a trust crisis warning signal is triggered; The behaviors include: successful collaboration and failed collaboration; if and and If so, the behavior is judged to be abnormal.

[0011] Preferably, the step of coordinating and scheduling target environmental entities and environmental resources through a collaborative engine according to a preset command scheme, and performing cross-domain trust transfer, includes: Define the interaction rules between entities based on the preset command plan; Based on the preset command plan, the constrained programming algorithm CP is used to dynamically allocate combat resources; Construct a cross-domain trust transfer model to facilitate trust transfer between multiple domains within a target region, including: Constructing a domain authority assessment model: ;in, Indicates the first Trust levels between this field and other fields; Indicates the number of domains; Indicates the evaluation coefficient; Constructing a trust transmission model: ;in, This indicates the level of trust between domain A and domain B; This indicates the level of trust in the direct interaction between domain A and domain B. Representation of domain agent Trust level with Domain B; This represents the direct empirical weight.

[0012] Preferably, the real-time deduction recapitulates and predicts the target area situation, and evaluates the robustness of the command scheme, including: By automatically marking key nodes, node data is processed using the data stream processing algorithm Kafka to generate tactical KPIs; Based on node data, a pre-trained LSTM network is used to predict the state of key nodes in the target area at a future time; The node data includes target entity aggregation time points, aggregation quantities, and motion states; the tactical KPIs include kill chain timeliness and late supply efficiency; Convert trust into risk weight in Monte Carlo deduction to evaluate the robustness of the command scheme, including: Construct a trust-aware reinforcement learning model: ; wherein, represents the trust of the agent; represents the risk sensitivity coefficient (when , low-trust partners are avoided), represents the discount factor; represents the learning rate; represents the immediate reward; represents the current Q value of performing action in state ; represents the Q value of performing action in the next state ; When , the system automatically avoids key decisions that rely on the agent; When , the system assigns core tasks to the agent to improve the collaboration efficiency of the agent and the robustness of the command scheme execution relying on the agent.

[0013] Preferably, when the deduction scenario changes, the trust of multiple agents is optimized, including: Extract the current scene environment features from the geographic information system, including terrain elevation distribution features; Obtain historical scene environment features from the database; Detect the KL divergence of environmental feature changes, i.e. ; wherein, represents the current scene environment feature distribution; represents the historical scene environment feature distribution; represents the scene change KL divergence, when When the scenario changes, it indicates that the trust migration needs to be performed for scenario replacement, including: The cross-scenario trust mapping function is established: wherein, represents the trust inheritance weight; represents the scenario sensitivity coefficient; represents the trust value of the original scenario; represents the trust value of the target scenario; represents the reference trust value of the target scenario; represents the scenario difference degree, wherein, represents the height vector of the target region in the original scenario; represents the height vector of the target region in the target scenario; The capability matching degree of the agent is corrected, that is: trust correction amount wherein, represents the capability score of the agent in the k type task; the task weight of the k type task in the new scenario; represents the capability correction coefficient (0.1-0.3); represents the length of time from the last capability verification; represents the number of task types; The scenario-behavior matrix is constructed :

[0014] Based on the scenario-behavior matrix, the scenario-based reference is provided for the fraud behavior monitor in the current scenario, specifically: The fraud behavior monitor calls the corresponding row vector in the scenario-behavior matrix as the reasonable threshold of the corresponding behavior; The deviation degree of the behavior is obtained, if , it is considered that the behavior is normal, and the scenario-behavior matrix is updated, and the updated scenario-behavior matrix element wherein, represents the update coefficient; ; if , it is considered that the behavior is abnormal, and the scenario-behavior matrix update is stopped.

[0015] Preferably, when the scenario changes, the multi-agent trust is optimized, and further includes: In the dynamic scenario, reinforcement learning is introduced for trust prediction, to improve the trust prediction accuracy and anti-interference ability, including: The state vector is set; wherein, a trust degree change time series representing the recent k-step behavior; a trust degree representing the agent at the current moment; a scene feature vector, a scene capability matching degree; A reinforcement learning model is constructed by combining a soft actor-critic algorithm SAC and a recurrent neural network RNN. The processed state vector is input into the reinforcement learning model, and the SAC outputs the action and the corresponding Q value, and the RNN network outputs the predicted trust degree.

[0016] A computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the multi-agent reinforcement learning-based distributed joint operation deduction engine architecture described above.

[0017] The beneficial effects of the present application are: 1. In the present application, in the case of changing scenes or sudden scenes, trust migration and sudden scene trust emergency handling are used to adapt to the trust problem among multiple agents in a dynamic scene, and ensure the cooperation efficiency among agents during scene switching.

[0018] 2. In the present application, reinforcement learning is introduced for trust degree prediction in a dynamic scene, which can significantly improve the accuracy and anti-interference ability of trust degree prediction. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The present application is a multi-agent reinforcement learning-based distributed joint operation deduction engine architecture block diagram. DETAILED DESCRIPTION

[0020] The following description is used to disclose the present application so that those skilled in the art can implement the present application. The preferred embodiments in the following description are only used as examples, and other obvious modifications can be thought of by those skilled in the art. The basic principles of the present application defined in the following description can be applied to other embodiments, modifications, improvements, equivalents and other technical solutions without departing from the spirit and scope of the present application.

[0021] It can be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of one element can be one, and in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number.

[0022] Embodiment one: Reference Figure 1The technical solution provided by this invention is: a distributed joint operation simulation engine architecture based on multi-agent reinforcement learning, comprising: a data processing layer, a simulation engine layer, a joint command layer, and an analysis and evaluation layer.

[0023] The data processing module is configured to acquire target area environmental data, entity historical state data and trust data, process target area environmental data and entity historical state data through a distributed computing framework, and generate an original trust vector using the trust data. The simulation engine layer is configured to perform environmental and entity modeling through an entity modeling engine and an environment simulation engine; and to perform dynamic trust modeling based on the agent's historical behavior. The joint command layer is configured to coordinate and schedule target environmental entities and environmental resources through a collaborative engine according to a preset command plan, and to perform cross-domain trust transmission. The analysis and evaluation layer is configured to be able to perform real-time deductions and replays and predict the situation in the target area, and to evaluate the robustness of the command plan.

[0024] The acquisition of environmental data, historical entity status data, and trust data for the target area can be achieved through the following steps: Low-latency target area environmental data and entity historical status data are obtained through the Data Distribution Service (DDS). Trust data is collected in real time through the DDS middleware. The trust data includes the communication response latency, task completion rate and resource utilization rate of the preset intelligent agents in the data processing layer. Building a trust dataset ;in, Indicates the first Communication response latency, task completion rate, and resource utilization of each agent.

[0025] The process of processing target area environmental data and entity historical status data using a distributed computing framework, and generating an original trust vector using trust data, can be achieved through the following steps: The database stores battlefield environment data, entity historical status data, and trust data. HPC or cloud computing is used to process environmental data and historical status data of entities in the target area. Determining the weight of trust data using the entropy weight method ;in, Indicates the first The entropy value of the trust data; Indicates the first The first agent of the intelligent agent Normalized values ​​of trust data; Indicates the number of agents; Indicates the number of elements in the trust dataset; Generating the original trust vector: ; wherein, represents the i-th agent trust vector.

[0026] Wherein, through the entity modeling engine, the environment simulation engine for environment modeling and entity modeling, the following steps can be implemented: Simulate the motion state of the entity through the physical model and the behavior model; Simulate real-time weather state and electromagnetic interference state through the dynamic environment model; Through the geographic information system, provide ground elevation data, and pass data basis for path planning and line of sight calculation.

[0027] The entity modeling engine includes a physical model and a behavior model, the physical model includes a sensor probability model and a ballistics equation; the behavior model includes a finite state machine FSM and a belief-desire-intention BDI decision model; The environment simulation engine includes a dynamic environment model and a geographic information system.

[0028] Wherein, based on the historical behavior of the agent, the trust degree is dynamically modeled, including the following steps: Based on the agent trust data, a trust decay model is constructed to solve the problem of decreasing credibility of historical trust data, specifically including: Constructing a trust decay model, that is, Let the trust degree of the current trust data be ; Wherein, represents the number of successful cooperation between agents (such as successful fire support response), represents the number of failed cooperation between agents (such as target missed), represents the decay coefficient, represents the time interval from the last exchange; Introducing a fraud behavior monitor to monitor the deviation degree of abnormal behavior, including: The fraud behavior monitor is ; wherein, represents the actual value of the behavior, represents the reasonable threshold of the behavior, represents the number of historical cooperation failures; represents the corrected number of cooperation failures, represents the deviation degree of the behavior, when , a trust crisis alarm signal is triggered; The behavior includes: cooperation success and cooperation failure; If and​ and If so, the behavior is judged to be abnormal.

[0029] The process of coordinating and scheduling target environmental entities and resources through a collaborative engine, and performing cross-domain trust transfer, according to a pre-defined command plan, can be achieved through the following steps: Define the interaction rules between entities based on the preset command scheme (e.g., the damage algorithm of anti-ship missiles to ships). Based on the preset command plan, the constrained programming algorithm CP is used to dynamically allocate combat resources; Construct a cross-domain trust transfer model to facilitate trust transfer between multiple domains within a target region, including: Constructing a domain authority assessment model: ;in, Indicates the first Trust levels between this field and other fields (based on historical data). Indicates the number of domains; Indicates the evaluation coefficient; Constructing a trust transmission model: ;in, This indicates the level of trust between domain A and domain B; This indicates the level of trust in the direct interaction between domain A and domain B. Representation of domain agent Trust level with Domain B; This represents the direct empirical weight (usually 0.6).

[0030] The real-time simulation and recap of the situation in the target area, and the assessment of the robustness of the command plan, can be achieved through the following steps: By automatically marking key nodes and using the Kafka data stream processing algorithm to process node data, tactical KPIs are generated. Based on node data, a pre-trained LSTM network is used to predict the state of key nodes in the target area at a future time. The node data includes the target entity aggregation time point, aggregation quantity, and movement status; the tactical KPIs include kill chain timeliness and post-resupply efficiency. Trust levels are converted into risk weights in Monte Carlo simulations to assess the robustness of command strategies, including: Constructing a trust-aware reinforcement learning model: ;in, This indicates the level of trust in the agent. Indicates the risk sensitivity coefficient (when (At that time, low-trust partners are avoided). represents a discount factor; represents a learning rate; represents an immediate reward; represents the current Q value of performing an action in the next state ; represents the Q value of performing an action in the next state ; when , the system automatically avoids relying on the key decisions of the agent; when , the system assigns the core task to the agent, improves the collaboration efficiency of the agent, and relies on the agent to improve the robustness of the command scheme execution.

[0031] Embodiment two: If the scenario changes or an emergency occurs, the original agent trust degree calculation and evaluation may not adapt to the new scenario. Therefore, on the basis of embodiment one, the following technical solutions are proposed: When the scenario changes, the multi-agent trust degree is optimized, including: Extracting the current scenario environment features from the geographic information system, including the terrain elevation distribution features; Obtaining historical scenario environment features from the database; Detecting the KL divergence of environmental feature changes, that is: ; wherein, represents the current scenario environment feature distribution; represents the historical scenario environment feature distribution; represents the scenario change KL divergence, when , it means that the scenario changes and needs to be replaced with trust migration, including: Establishing a cross-scenario trust mapping function: ; wherein, represents the trust inheritance weight (default 0.4); represents the scene sensitivity coefficient (default 0.8); represents the trust degree value of the original scene; represents the trust degree value of the target scene; represents the target scene reference trust value (city 0.6, mountain 0.75); represents the scene difference degree, ; wherein, represents the elevation vector of the target area in the original scene; represents the elevation vector of the target area in the target scene; The agent capability matching degree is corrected, that is: Trust degree correction amount ; wherein, represents the capability score of the agent in the k type task; the task weight of the k type task in the new scene; represents the capability correction coefficient (0.1-0.3); represents the length of time from the last capability verification; represents the number of task types; Construct a scene-behavior matrix ; ; for example, represents that the reasonable threshold range of the communication frequency of the unmanned aerial vehicle in the city scene is [0.2, 0.5].

[0032] Based on the scene-behavior matrix, a scene-based benchmark is provided for the fraud behavior monitor in the current scene, specifically: The fraud behavior monitor calls the corresponding row vector in the scene-behavior matrix as the corresponding behavior reasonable threshold; Obtain the deviation degree of the behavior, if , it is considered that the behavior is normal, and the scene-behavior matrix is updated, and the updated scene-behavior matrix element ; wherein, represents the update coefficient; ; If , it is considered that the behavior is abnormal, and the scene-behavior matrix update is stopped.

[0033] For example, in the trust maintenance of an amphibious landing operation unmanned ship cluster. Load the initial scene-behavior matrix, the communication frequency ; the speed ; Unmanned ship A behavior: communication frequency is 0.15Hz (lower than the threshold); speed 28 knots (exceeds the threshold); deviation degree of communication frequency ; the deviation degree of the speed ; are greater than 0.3; the fraud behavior monitor marks A as a suspicious unit; the scene-behavior matrix refuses to update this data (to avoid polluting the baseline); It is found that the data communication delay time of the ship exceeds the normal range, and it is judged that the ship may be interfered, hijacked or lured.

[0034] Previously, LSTM has been used for trust prediction, and in the following, a reinforcement learning (RL) algorithm is introduced, which can optimize the prediction process. The specific scheme is as follows: Introduce reinforcement learning for trust prediction in dynamic scenarios to improve trust prediction accuracy and anti-interference ability, including the following steps: Set the state vector ; wherein, represents the recent k-step trust degree change time series; represents the trust degree of the agent at the current moment; represents the scene feature vector, represents the scene capability matching degree; For example, the state vector in the urban alleyway scene is: [0.75, [0.02, -0.01, 0.03], [0.8, 0.3, 0.6], [0.8, 0.85], 0.78].

[0035] Wherein, [0.02, -0.01, 0.03] represents the recent 3-step trust degree change time series, and [0.8, 0.3, 0.6] represents the elevation change vector. A reinforcement learning model is constructed by combining a soft actor-critic algorithm SAC and a recurrent neural network RNN; The processed state vector is input into the reinforcement learning model, the action and the corresponding Q value are output by the SAC, and the predicted trust degree is output by the RNN network; For example, in the unmanned aerial vehicle cluster coordination scene: Initial state: the trust degree of the unmanned aerial vehicle is 0.8; the scene is a regular scene (urban reconnaissance); Emergency event: electromagnetic interference causes communication terminal; The reinforcement learning model monitors the scene change and outputs the action a (communication through the backup channel), and the predicted trust degree is 0.77; After the communication is re-established, the trust degree of the unmanned aerial vehicle is updated.

[0036] The application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the functions of the distributed joint operation deduction engine architecture based on multi-agent reinforcement learning.

[0037] The processes described above with reference to the flowcharts can be implemented as computer software programs in accordance with embodiments of the present disclosure. Embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication section, and / or installed from a detachable medium. When the computer program is executed by a central processing unit (CPU), the above-described functions defined in the methods of the present application are performed. It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take a variety of forms, including but not limited to, an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer the program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to, wireless, wire, optical cable, RF or the like, or any suitable combination of the above.

[0038] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0039] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functions and structural principles of the present invention have been shown and explained in the embodiments. Without departing from the principles described, the implementation of the present invention may have any changes or modifications.

Claims

1. A distributed joint combat simulation engine architecture based on multi-agent reinforcement learning, characterized in that, include: The data processing layer, wherein the data processing module is configured to acquire target area environmental data, entity historical state data and trust data, process the target area environmental data and entity historical state data through a distributed computing framework, and generate an original trust vector using the trust data; A simulation engine layer, configured to perform environment modeling and entity modeling through an entity modeling engine and an environment simulation engine; And based on the agent's historical behavior, dynamic trust level modeling is performed; The joint command layer is configured to coordinate and schedule target environmental entities and environmental resources through a collaborative engine according to a preset command plan, and to perform cross-domain trust transmission. An analysis and evaluation layer is configured to perform real-time deductions and replays, predict the situation in the target area, and assess the robustness of command plans.

2. The distributed joint combat simulation engine architecture based on multi-agent reinforcement learning as described in claim 1, characterized in that, The acquisition of target area environmental data, entity historical status data, and trust data includes: Low-latency target area environmental data and entity historical status data are obtained through the Data Distribution Service (DDS). Trust data is collected in real time through the DDS middleware. The trust data includes the communication response latency, task completion rate and resource utilization rate of the preset intelligent agents in the data processing layer. Building a trust dataset ;in, Indicates the first Communication response latency, task completion rate, and resource utilization of each agent.

3. The distributed joint combat simulation engine architecture based on multi-agent reinforcement learning as described in claim 2, characterized in that, The process of processing target area environmental data and entity historical state data through a distributed computing framework, and generating an original trust vector using trust data, includes: The database stores battlefield environment data, entity historical status data, and trust data. HPC or cloud computing is used to process environmental data and historical status data of entities in the target area. Determining the weight of trust data using the entropy weight method ;in, Indicates the first The entropy value of the trust data; Indicates the first The first agent of the intelligent agent Normalized values ​​of trust data; Indicates the number of agents; Indicates the number of elements in the trust dataset; Generate the original trust vector: ;in, Indicates the first Trust vectors for each agent.

4. The distributed joint combat simulation engine architecture based on multi-agent reinforcement learning as described in claim 3, characterized in that, The process of performing environmental and entity modeling using a solid modeling engine and an environment simulation engine includes: The entity modeling engine includes a physical model and a behavioral model. The physical model includes a sensor probability model and ballistic equations. The behavioral model includes a finite state machine (FSM) and a belief-expectation-intention (BDI) decision model. The environment simulation engine includes a dynamic environment model and a geographic information system; Simulate the motion state of entities using physical and behavioral models; Real-time weather and electromagnetic interference conditions are simulated using a dynamic environment model. Geographic Information Systems (GIS) provide ground elevation data, serving as a data foundation for route planning and sightline calculations.

5. The distributed joint combat simulation engine architecture based on multi-agent reinforcement learning according to claim 4, characterized in that, The dynamic trust modeling based on the agent's historical behavior includes: Based on agent trust data, a time-based trust decay model is constructed to address the issue of declining credibility of historical trust data; including: Construct a trust decay model, namely: Let the trust level of the current trust data be set. ; in, This represents the number of successful collaborations between intelligent agents. This indicates the number of failed collaborations between intelligent agents. Indicates the attenuation coefficient. Indicates the time interval since the last exchange; Introduce a deceptive behavior monitor to track deviations from abnormal behaviors, including: Deceptive behavior monitor ;in, Indicates the actual value of the behavior. Indicates a reasonable threshold for behavior. Indicates the number of historical collaboration failures; This indicates the number of collaboration failures after the correction. Indicates the degree of deviation from the behavior, when At that time, a trust crisis warning signal is triggered; The behaviors include: successful collaboration and failed collaboration; if and and If so, the behavior is judged to be abnormal.

6. The distributed joint combat simulation engine architecture based on multi-agent reinforcement learning according to claim 5, characterized in that, The process of coordinating and scheduling target environmental entities and resources through a collaborative engine according to a preset command plan, and performing cross-domain trust transfer, includes: Define the interaction rules between entities based on the preset command plan; Based on the preset command plan, the constrained programming algorithm CP is used to dynamically allocate combat resources; Construct a cross-domain trust transfer model to facilitate trust transfer between multiple domains within a target region, including: Constructing a domain authority assessment model: ;in, Indicates the first Trust levels between this field and other fields; Indicates the number of domains; Indicates the evaluation coefficient; Constructing a trust transmission model: ;in, This indicates the level of trust between domain A and domain B; This indicates the level of trust in the direct interaction between domain A and domain B. Representation of domain Agent Trust level with domain B; This represents the direct empirical weight.

7. The distributed joint combat simulation engine architecture based on multi-agent reinforcement learning according to claim 6, characterized in that, The real-time simulation and replay of the situation in the target area, and the assessment of the robustness of the command plan, include: By automatically marking key nodes and using the Kafka data stream processing algorithm to process node data, tactical KPIs are generated. Based on node data, a pre-trained LSTM network is used to predict the state of key nodes in the target area at a future time. The node data includes the target entity aggregation time point, aggregation quantity, and movement status; the tactical KPIs include kill chain timeliness and post-resupply efficiency. Trust levels are converted into risk weights in Monte Carlo simulations to assess the robustness of command strategies, including: Constructing a trust-aware reinforcement learning model: ;in, This indicates the level of trust in the agent. Indicates the risk sensitivity coefficient (when (At that time, low-trust partners are avoided). Indicates the discount factor; Indicates the learning rate; Indicates an immediate reward; Indicates the state Next action The current Q value; Indicate the next state Execution of actions Q value; when At that time, the system automatically avoids critical decisions that rely on the agent; when At that time, the system assigns core tasks to the agent to improve the agent's collaboration efficiency and relies on the agent to improve the robustness of the command plan execution.

8. The distributed joint combat simulation engine architecture based on multi-agent reinforcement learning according to claim 7, characterized in that, When the simulation scenario changes, optimize the trust level of multi-agent agents, including: Extract current scene environmental features from the geographic information system, including terrain elevation distribution features; Retrieve historical scene and environmental features from the database; The KL divergence for detecting changes in environmental characteristics, i.e.: ;in, This indicates the distribution of environmental features in the current scene; Indicates the distribution of environmental features in historical scenes; Represents the KL divergence of scene changes, when When this occurs, it indicates a change in the simulation scenario, requiring a trust migration involving scenario replacement, including: Establish cross-scenario trust mapping function: ;in, Indicates the weight of trust inheritance; Indicates the scene sensitivity coefficient; This represents the trust level value in the original scenario; This indicates the trust level of the target scenario; This represents the baseline trust value for the target scenario; Indicates the degree of difference in scenarios. ;in, This represents the elevation vector of the target area in the original scene; This represents the elevation vector of the target area within the target scene. Adjust the agent's ability matching degree, that is: Trust level correction ;in, This represents the agent's performance score in k types of tasks; Task weights for k types of tasks in the new scenario; The capability correction factor is 0.1-0.

3. Indicates the length of time since the last capability verification; Indicates the number of task types; Building a scenario-behavior matrix : ； Based on the scenario-behavior matrix, a scenario-based benchmark is provided for the deception behavior monitor in the current scenario, specifically: The deception behavior monitor calls the corresponding row vector in the behavior matrix as a reasonable threshold for the corresponding behavior. Obtain the deviation of the behavior, if If the behavior is normal, the scene-behavior matrix is ​​updated, and the updated scene-behavior matrix elements are... ;in, Indicates the update coefficients; ; if If the behavior is abnormal, the scene-behavior matrix update is stopped.

9. The distributed joint combat simulation engine architecture based on multi-agent reinforcement learning according to claim 8, characterized in that, Optimizing multi-agent trust when the simulation scenario changes also includes: Introducing reinforcement learning for trust prediction in dynamic scenarios improves the accuracy and robustness of trust prediction, including: Set the state vector ;in, This represents the time series of trust changes based on the behavior of the most recent k steps; This indicates the agent's trust level at the current moment; Represents the scene feature vector. Indicates the matching degree of scene capabilities; A reinforcement learning model is constructed by combining the soft actor-critic algorithm (SAC) with a recurrent neural network (RNN). The preprocessed state vector is input into the reinforcement learning model. The SAC outputs the action and the corresponding Q value, and the RNN network outputs the predicted confidence level.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the distributed joint combat simulation engine architecture based on multi-agent reinforcement learning as described in any one of claims 1-9.

Images