An intelligent game training and simulation device for complex equipment system confrontation

Through the unified interface of microservice architecture and multi-grained model, the problems of high coupling degree and limited performance in the intelligent game adversarial system are solved, and complex combat mission simulation and DRL model are realized, and multi-agent training is supported. and unified interface specifications improve the stability and scalability of the system.

CN115394140BActive Publication Date: 2024-11-15BEIJING CHUANGQI VISION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210999053.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-19
Publication Date
2024-11-15
Estimated Expiration
2042-08-19

AI Technical Summary

Technical Problem

The existing intelligent game adversarial system has problems such as high coupling of functional modules, difficulty in maintaining codes, limited performance, failure to support deep reinforcement learning of multiple agents, and lack of unified DRL algorithm interface specifications, making it difficult to achieve rapid simulation and algorithms for complex combat tasks. Build.

Method used

Adopt microservice architecture design, including simulation basic environment and intelligent training framework, supports a unified interface for multi-grained model, and has built-in multiple DRL algorithm components to realize independent development and deployment of functional services, supports single and multi-agent training methods, and provides unified Interface and message middleware interact with data.

Benefits of technology

It realizes clear logic, high stability and high scalability, supports complex combat mission simulation, reduces the workload of algorithm development, improves the rapid development and iteration capabilities of DRL models, and realizes seamless integration between DRL models and simulation systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115394140B_ABST
    Figure CN115394140B_ABST
Patent Text Reader

Abstract

The present invention discloses an intelligent game training and simulation device for complex equipment system confrontation, a simulation basic environment and an intelligent training framework designed with a microservice architecture, the simulation basic environment is used to realize the simulation of complex equipment system confrontation, and the intelligent training framework interacts with the simulation basic environment data to realize the intelligent game training of complex equipment system confrontation, solves the architectural design problem of the intelligent game training and simulation device, effectively utilizes computing resources, and supports complex campaign-level combat mission simulation; solves the unified interface specification problem of the DRL algorithm, and supports the training methods of SADRL and MADRL at the same time, and quickly constructs new algorithms based on the dynamic combination of DRL algorithm components, reduces the workload of modeling personnel, and realizes the rapid development and iteration of the DRL model; simplifies the data interaction between the DRL model and the simulation system, solves the deployment of the DRL model and the integration problem with the simulation system, and forms an efficient CGF intelligent behavior modeling overall process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of system modeling and simulation, and in particular to an intelligent game training and simulation device for complex equipment system confrontation. Background Art

[0002] System simulation is an effective means to solve many military problems under the current information conditions. Computer Generated Force (CGF) in the simulation system can use computer programs to replace personnel and equipment in the system to participate in the simulation operation, greatly reducing the demand for personnel and simulation equipment, helping to expand the scale of simulation and effectively reduce costs. The generated virtual forces are flexible and diverse, making flexible control and prediction of the simulation possible.

[0003] One of the core tasks of building CGF is to model the behavior of combat entities in battlefield environments. Traditional behavior modeling methods include rule sets and finite state machines, which have the disadvantages of difficulty in acquiring domain knowledge, large modeling workload, and low efficiency. In addition, the generated behaviors are fixed and lack adaptability. With the high degree of informatization of modern warfare and the high intelligence of combat equipment, research on CGF intelligent behavior modeling has attracted more and more attention. In recent years, the application of deep reinforcement learning (DRL) methods to the field of CGF intelligent behavior modeling has achieved outstanding results. The DRL-based model has the advantages of low domain knowledge requirements, high modeling efficiency, and strong generalization ability.

[0004] In the process of using DRL to model CGF intelligent behavior, a large amount of data needs to be interacted with the simulation system, including situation information on the battlefield environment, decision information calculated by neural network, control information of the simulation process, etc. These data interactions are often complex, cumbersome, and repetitive. Providing a unified framework and interface to support the above interaction process simplifies data processing related operations and helps improve modeling efficiency. At the same time, the existing DRL algorithms are constantly being updated. Modelers are usually not related professional researchers and are not sure about the latest progress of these algorithms. An intelligent game training and simulation device is provided for complex equipment combat systems. It has a built-in DRL algorithm with the best performance at present, so that modelers can directly apply existing algorithms, focus on the process of CGF intelligent behavior modeling, and realize the rapid development and iteration of CGF models.

[0005] Existing intelligent game confrontation systems mostly exist in the form of a single architecture, divided into a simulation and deduction platform and an AI confrontation training management system. The simulation and deduction platform generates combat situation data and receives combat decision data; the AI ​​confrontation training management system is responsible for the development and training of the DRL model, as well as the start and stop and operation control of the simulation and deduction platform. Such a platform has the following problems:

[0006] First, the degree of coupling between the functional modules of the monolithic application is high, and the maintenance, reconstruction and deployment of the code are relatively difficult. The stability and scalability are not high. At the same time, it will be limited by the performance of a single computer, making it difficult to complete complex combat mission simulations with a large number of simulation entities and high resource consumption, and the platform capabilities are limited;

[0007] Second, the support for DRL algorithms is low. Only a small number of algorithms are implemented and only single-agent deep reinforcement learning (SADRL) is supported. The training method of multi-agent deep reinforcement learning (MADRL) is not supported. In addition, a unified DRL algorithm interface specification has not been formed, and the logical decomposition of each component of the DRL algorithm has not been achieved. The function of quickly building an algorithm that is more suitable for combat missions based on the dynamic combination of DRL algorithm components cannot be realized. The workload of CGF modelers in algorithm development is heavy;

[0008] Third, it only supports a single operating mode with an AI adversarial training management system as the main system and a simulation deduction platform as the auxiliary system. It does not consider the deployment of the DRL model and its integration with the simulation system. This is not conducive to conducting research on equipment deployment optimization, weapon effectiveness evaluation, and other research for the combat system after the model training is completed. Summary of the invention

[0009] The present invention aims to provide an intelligent game training and simulation device for complex equipment system confrontation to solve or improve at least one of the above-mentioned technical problems.

[0010] In view of this, a first aspect of the present invention is to provide an intelligent game training and simulation device for complex equipment system confrontation.

[0011] The first aspect of the present invention provides an intelligent game training and simulation device for complex equipment system confrontation, including: a simulation basic environment and an intelligent training framework designed with a microservice architecture, wherein the simulation basic environment is used to realize the simulation of the complex equipment system confrontation, and the intelligent training framework interacts with the simulation basic environment data to realize the intelligent game training of the complex equipment system confrontation; the intelligent training framework includes: model reasoning, experience storage and intelligent optimization, and the simulation basic environment includes: a simulation engine and a CGF intelligent behavior model; the microservice architecture includes: a service layer, and the service layer includes an intelligent reasoning service for the model reasoning, an intelligent storage service for the experience storage, an intelligent optimization service for network optimization, and a simulation engine service for the simulation engine; wherein the intelligent reasoning service, the intelligent storage service, and the intelligent optimization service all adopt a unified external interface, and each has a plurality of DRL algorithm components built in, and the simulation engine adopts a multi-granularity model unified interface.

[0012] Furthermore, the intelligent game training and simulation device for complex equipment system confrontation also includes: an intelligent confrontation learning training interface, which is configured to receive and send message middleware to realize data interaction between the intelligent training framework and the simulation basic environment; wherein, the message middleware includes: simulation parameters, system status, situation information and behavior decisions.

[0013] Furthermore, the intelligent adversarial learning training interface includes: a parameter configuration interface for receiving and sending the simulation parameters; a state monitoring interface for receiving and sending the system state; a situation acquisition interface for receiving and sending the situation information; and a behavior execution interface for receiving and sending the behavior decision.

[0014] Furthermore, the service layer can be centrally deployed on a single computer or distributed on multiple computers, and the service layer also includes: scenario generation service, experiment design service, model management service, performance evaluation service, simulation control service and simulation data service for the simulation basic environment.

[0015] Furthermore, the simulation engine service includes at least one of the following: subject management, model-driven scenario analysis, simulation control, time management and data interaction.

[0016] Furthermore, the microservice architecture also includes: a platform layer, which is used to provide underlying support for system maintenance management, container management and orchestration; a resource layer, which is used to abstract various types of data resources and provide persistent storage and basic management operations for resources; a gateway layer, which is a unified entrance for the platform, routes API requests, and can perform identity authentication and resource caching; a tool layer, which is used to provide a human-computer interaction interface.

[0017] Furthermore, the platform layer adopts a cloud operating system and is specifically used to implement at least one of the following functions: virtualization, container service, service discovery, storage orchestration, automatic resource scheduling, cluster management and monitoring, log collection and storage, and service definition service deployment.

[0018] Furthermore, the resource layer is specifically used to implement at least one of the following functions: simulation model management, data management, scenario management, document management, data type management, map service, situation-related resource management and intelligent resource management.

[0019] Furthermore, the gateway layer is specifically used to implement at least one of the following functions: identity authentication, load balancing, caching and monitoring.

[0020] Furthermore, the tool layer specifically includes at least one of the following: simulation control tools, scenario editing tools, intelligent training tools, two-dimensional and three-dimensional situation tools, model management tools, experimental design tools, performance evaluation tools and data recording and playback tools.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The microservice architecture is introduced into the architecture design of the intelligent game training and simulation device, which realizes the independent development and deployment of functional services, loose coupling, and technical heterogeneity, so that the intelligent game training and simulation device has the advantages of clear logic, high stability, and high scalability. The simulation engine supports the distributed simulation of multi-source heterogeneous models by implementing the unified interface specification of multi-granularity models, and has the ability to simulate complex campaign-level combat tasks;

[0023] The DRL model training process is divided into three sub-processes: model reasoning, experience storage, and network optimization. They are deployed as three services, each of which provides a unified interface to the outside world. It has multiple DRL algorithm components built in, supports both SADRL and MADRL training methods, realizes the dynamic combination of DRL algorithm components, and supports the creation and uploading of custom algorithms or algorithm components, which reduces the workload of algorithm development, improves the efficiency of DRL model training, and has the ability to quickly develop and iterate DRL models.

[0024] By supporting both model training and independent operation modes, it provides support for the entire process from model training to final deployment. The intelligent reasoning service uses the same set of interfaces in different modes, realizing the seamless integration of the DRL model and the simulation system, incorporating the CGF intelligent behavior model into the simulation system, and improving the simulation and deduction functions in complex equipment system confrontation scenarios, providing an effective theoretical basis and technical assistance for research on optimization of combat command strategies and improvement of combat levels.

[0025] Additional aspects and advantages of embodiments according to the present invention will become apparent in the following description or may be learned through practice of embodiments according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0027] Figure 1 It is a schematic diagram of the functional composition of the intelligent game training and simulation device of the present invention;

[0028] Figure 2 A schematic diagram of the software architecture of the intelligent game training and simulation device of the present invention;

[0029] Figure 3 A schematic diagram of data interaction details of the intelligent game training and simulation device of the present invention;

[0030] Figure 4 It is the overall operation flow chart of the intelligent game training and simulation device of the present invention;

[0031] Figure 5 This is a specific flow chart of the model control training of the present invention;

[0032] Figure 6 It is a specific flow chart of the control simulation of the present invention. DETAILED DESCRIPTION

[0033] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0035] See also Figure 1-6The first aspect of the present invention provides an intelligent game training and simulation device for complex equipment system confrontation, including: a simulation basic environment and an intelligent training framework designed with a microservice architecture, the simulation basic environment is used to realize the simulation of complex equipment system confrontation, and the intelligent training framework interacts with the simulation basic environment data to realize the intelligent game training of complex equipment system confrontation; the intelligent training framework includes: model reasoning, experience storage and intelligent optimization, and the simulation basic environment includes: a simulation engine and a CGF intelligent behavior model; the microservice architecture includes: a service layer, the service layer includes an intelligent reasoning service for model reasoning, an intelligent storage service for experience storage, an intelligent optimization service for network optimization, and a simulation engine service for the simulation engine; wherein the intelligent reasoning service, the intelligent storage service, and the intelligent optimization service all adopt a unified external interface, and each has a plurality of DRL algorithm components built in, and the simulation engine adopts a multi-granularity model unified interface.

[0036] The present invention introduces microservice architecture into the architectural design of intelligent game training and simulation devices, realizes independent development and deployment of functional services, loose coupling, and technical heterogeneity, so that the intelligent game training and simulation devices have the advantages of clear logic, high stability, and high scalability. The simulation engine supports distributed simulation of multi-source heterogeneous models by implementing a unified interface specification for multi-granularity models, and has the ability to simulate complex campaign-level combat missions, solves the architectural design problems of intelligent game training and simulation devices, effectively utilizes computing resources, and supports complex campaign-level combat mission simulations.

[0037] The DRL model training process is split into three sub-processes: model reasoning, experience storage, and network optimization. They are deployed as three services, each of which provides a unified interface to the outside world. It has built-in multiple DRL algorithm components and supports both SADRL and MADRL training methods. It realizes the dynamic combination of DRL algorithm components and supports the creation and uploading of custom algorithms or algorithm components, which reduces the workload of algorithm development and improves the efficiency of DRL model training. It has the ability to quickly develop and iterate DRL models. This framework can meet the training process of SADRL and support the training method of MADRL. It uses the centralized training and distributed execution methods of MADRL. Taking the DRL method as an example, it is only necessary to deploy DRL model parameters and set exploration strategies for multiple agents in the intelligent reasoning service, jointly store the MDP data information of all agents in the intelligent storage service, and implement the logical process of centralized training in the intelligent optimization service. The interaction interface between services remains unchanged; at the same time, this framework can also realize the dynamic combination of DRL algorithm components. For example: in the basic DDPG algorithm, ordinary experience replay or priority experience replay technology can be used at will according to the user's wishes, and Gaussian white noise or Ornstein-Uhlenbeck noise can be used as exploration noise to form behavioral strategies. In addition, if the modeler wants to implement a new DRL algorithm or algorithm component, he only needs to implement the interface required by the corresponding service and upload it, thereby reducing the workload of algorithm development and having the ability to quickly develop and iterate the DRL model;

[0038] By supporting model training and independent operation modes, the whole process from model training to final deployment is supported. The intelligent reasoning service uses the same set of interfaces in different modes, realizing the seamless integration of the DRL model and the simulation system, incorporating the CGF intelligent behavior model into the simulation system, and improving the simulation function in the complex equipment system confrontation scenario. It provides an effective theoretical basis and technical assistance for the optimization of combat command strategies and the improvement of combat levels. The intelligent game training and simulation device supports two modes: model training and independent operation. In the model training mode, the user configures the simulation parameters, specifies the MDP data structure to be stored, and configures the DRL algorithm parameters on the intelligent training tool interaction interface to start the model training process; in the independent operation mode, the user deploys the trained DRL model to the intelligent reasoning service, closes the exploration strategy of the intelligent reasoning service, and uses the simulation control tool to start the simulation process. The intelligent reasoning service and the intelligent agent model constitute an important bridge between the simulation environment and the intelligent training framework. The intelligent reasoning service provides a decision-making calculation interface. The intelligent agent model and the force simulation model are assembled to form an intelligent force model, collect the situation information of the force simulation model, and call the interface of the intelligent reasoning service to obtain and execute the behavior decision. The intelligent game training and simulation device provides support for the entire process from model training to final deployment through two modes, realizing the seamless integration of the DRL model and the simulation system.

[0039] Specifically, model reasoning corresponds to the intelligent reasoning service, which is responsible for calculating the original decision using the target strategy, obtaining the final decision based on the exploration strategy, and interacting with the environment. The intelligent reasoning service provides interfaces such as calculating behavioral decisions based on situation information, updating neural network structures and parameters, and configuring behavioral strategy exploration parameters, and internally implements different types of exploration noise;

[0040] Experience storage corresponds to the intelligent storage service, which is responsible for storing the Markov decision process (MDP) experience data generated by interacting with the environment. The intelligent storage service provides interfaces such as storing experience data, obtaining experience data, and configuring experience playback pool parameters, and implements different experience pool technologies internally.

[0041] Network optimization corresponds to the intelligent optimization service, which is responsible for optimizing the parameters of the target strategy neural network using empirical data. The intelligent optimization service provides interfaces such as configuration of algorithm types and neural network hyperparameters, training process control and monitoring, and implements different algorithm optimization methods internally.

[0042] Furthermore, an intelligent game training and simulation device for complex equipment system confrontation also includes: an intelligent confrontation learning training interface, which is configured to receive and send message middleware to realize data interaction between the intelligent training framework and the simulation basic environment; wherein, the message middleware includes: simulation parameters, system status, situation information and behavior decisions.

[0043] Furthermore, the intelligent adversarial learning training interface includes: a parameter configuration interface for receiving and sending simulation parameters; a state monitoring interface for receiving and sending system state; a situation acquisition interface for receiving and sending situation information; and a behavior execution interface for receiving and sending behavior decisions.

[0044] Furthermore, the service layer can be centrally deployed on a single computer or distributed on multiple computers. The service layer also includes: scenario generation services for simulation basic environment, experimental design services, model management services, performance evaluation services, simulation control services and simulation data services.

[0045] Specifically, when deployed in a distributed manner, services communicate through message middleware to complete complex simulation and intelligent training functions.

[0046] Specifically, the service layer has multiple core microservices that provide important functions such as simulation and intelligent training, and each service is responsible for:

[0047] ①Simulation engine service: The simulation engine service is used to implement model scheduling, data interaction between models, and various management functions around data interaction. It supports distributed simulation and realizes the orderly advancement of the simulation process.

[0048] ② Scenario generation service: Scenario generation service supports users to conveniently and flexibly deploy combat forces and formulate combat procedures, and realizes docking with simulation model system and simulation scenario description specifications through standardized interfaces;

[0049] ③ Experimental design service: The experimental design service is used to assist experimental designers to effectively complete experimental designs of various modes, generate experimental design sample space description files and provide them to the simulation engine for scheduling;

[0050] ④ Model management service: Model management service provides functions such as model interface definition, model template generation, combined model construction, model management operation, etc., providing users with indirect access to simulation model resources;

[0051] ⑤Effectiveness evaluation service: the effectiveness evaluation service enables users to carry out system capability index evaluation based on simulation test data, thus automating the index evaluation work;

[0052] ⑥Simulation control service: simulation control service supports simulation parameter settings such as simulation step length and simulation speed, as well as simulation process control operations such as simulation engine initialization, start and stop, and stepping;

[0053] ⑦Simulation data service: simulation data service provides functions such as acquisition, combination, distribution and storage of simulation data;

[0054] ⑧ Intelligent reasoning service: Intelligent reasoning service provides computing functions for obtaining behavioral decisions based on situation information, and supports different parameter configurations in two modes: model training and independent operation.

[0055] ⑨ Intelligent storage service: Intelligent storage service provides the function of storing MDP experience data generated by interaction with the environment, and supports custom reward shaping methods;

[0056] ⑩ Intelligent optimization service: The intelligent optimization service provides the function of optimizing the parameters of the target strategy neural network using empirical data, and has a built-in algorithm implementation with the best current performance (State Of The Art, SOTA).

[0057] The resource layer abstracts various data resources and provides support for persistent storage and basic management operations. Data resources are divided into two categories: simulation resources and intelligent resources. Simulation resources include force simulation models, combat scenario files, test design samples, simulation data records, effectiveness evaluation projects, etc. Intelligent resources include intelligent training tasks, DRL model parameters, etc. The data resources of the resource layer are stored in the database in the form of microservices, supporting basic operations such as create, read, update, and delete (Create, Read, Update, Delete, CRUD) for various resources.

[0058] Furthermore, the simulation engine service includes at least one of the following: subject management, model-driven scenario analysis, simulation control, time management and data interaction.

[0059] Furthermore, the microservice architecture also includes: a platform layer, which is used to provide underlying support for system maintenance management, container management and orchestration; a resource layer, which is used to abstract various types of data resources and provide persistent storage and basic management operations for resources; a gateway layer, which is a unified entrance for the platform, routes API requests, and can perform identity authentication and resource caching; and a tool layer, which is used to provide a human-computer interaction interface.

[0060] Furthermore, the platform layer adopts a cloud operating system and is specifically used to implement at least one of the following functions: virtualization, container service, service discovery, storage orchestration, automatic resource scheduling, cluster management and monitoring, log collection and storage, and service definition service deployment.

[0061] Specifically, the platform layer includes physical hardware facilities such as servers, data centers, firewalls, VPN gateways, switches, routers, etc., and provides underlying functions such as virtualization, container scheduling and orchestration, infrastructure automation, and system maintenance and management.

[0062] Furthermore, the resource layer is specifically used to implement at least one of the following functions: simulation model management, data management, scenario management, document management, data type management, map services, situation-related resource management and intelligent resource management.

[0063] Furthermore, the gateway layer is specifically used to implement at least one of the following functions: identity authentication, load balancing, caching and monitoring.

[0064] Specifically, the gateway layer serves as the unified entrance to the platform, routes and distributes API requests, and has capabilities such as identity authentication, load balancing, resource caching, and status monitoring. It encapsulates the internal details of the platform and optimizes the communication process between external clients and backend services.

[0065] Furthermore, the tool layer specifically includes at least one of the following: simulation control tools, scenario editing tools, intelligent training tools, two-dimensional and three-dimensional situation tools, model management tools, experimental design tools, performance evaluation tools, and data recording and playback tools.

[0066] Specifically, the tool layer provides a dedicated human-computer interaction UI interface for each service, calls various service functions, and reduces the difficulty of using intelligent game training and simulation devices. Among them, the simulation control tool selects simulation tasks and controls simulation processes; the model management tool manages simulation model resources; the scenario editing tool completes the development of scenarios; the experimental design tool is used for experimental design; the intelligent training tool supports training parameter configuration, process control and status monitoring; the training effectiveness evaluation tool builds an indicator system and calculates performance indicators; the two-dimensional and three-dimensional situation tool observes the simulation situation change process in real time; the data recording and playback tool configures the required data recording and selects the data to be played back.

[0067] Example 1

[0068] like Figure 3-6 The overall operation process is as shown in Figure 4 As shown, the specific steps include:

[0069] Step 1: Develop a force simulation model and an intelligent agent model. The intelligent agent model implements the logic of obtaining the required situation data and executing decision results, and is combined with the force simulation model to form an intelligent force model, which is uploaded to the model resource library.

[0070] Specific steps:

[0071] Step 101: Use the model management tool to define the interface parameters such as initialization, input, and output of the force simulation model, generate a force simulation model template, and complete the development;

[0072] Step 102: Generate an intelligent agent model template using a model management tool and complete the development, and implement the logic of obtaining the required situation data and executing the decision results in the intelligent agent model;

[0073] Step 103: Use the model management tool to combine the force simulation model and the intelligent agent model, generate an intelligent force model template, complete the development and upload it to the model resource library.

[0074] Step 2: Develop simulation scenarios, determine the combat area, deploy force entities, set combat mission processes, configure model interactions, and generate simulation scenario files.

[0075] Specific steps:

[0076] Step 201: Use the scenario editing tool to create a new scenario file, determine information such as battlefield areas and combat area elements, divide troop camps, deploy troop entities, set combat mission processes, configure DDS topic lists, model data publishing and subscription interactions, and generate simulation scenario XML files.

[0077] Step 3: Based on the simulation scenario file, select the experimental factors and experimental design methods, and generate the experimental design sample space file.

[0078] Specific steps:

[0079] Step 301: Use the experimental design tool to create a new experimental design plan based on the simulation scenario, select experimental factors, build an experimental design factor system, select an experimental design method, complete the experimental design, and generate an experimental design sample space file.

[0080] Step 4: In the model training mode, create an intelligent training task, configure the situation data and decision data that need to interact with the simulation system; develop and upload a custom DRL algorithm, or directly use the built-in DRL algorithm, configure the DRL algorithm parameters, simulation parameters, and training control parameters; perform intelligent training process control, and iterate until the model converges. The specific process of model training is as follows: Figure 5 shown.

[0081] Specific steps:

[0082] Step 401: Use the intelligent training tool to create an intelligent training task and manage the intelligent training process in a systematic manner. In the intelligent training task, configure the simulation scenario file and experimental design sample file to be used. The interactive interface will display all models and their parameters involved in the simulation scenario. Define the state space by checking the model output parameters and define the action space by checking the model input parameters. The intelligent agent model will use the data structure defined above to interact with the intelligent reasoning service. The simulation data service will automatically capture and assemble the relevant data to form an MDP structure and store it in the intelligent storage service;

[0083] Step 402: Determine whether a custom DRL algorithm needs to be developed. If so, develop and implement the corresponding component according to the DRL algorithm component interface of each service and upload it. Each service will dynamically identify the new component and make it available. Otherwise, jump to step 403;

[0084] Step 403: Use the intelligent training tool to select the DRL algorithm and components to be used. If the SADRL algorithm is selected, you only need to configure the parameters of a single SADRL model. If the MADRL algorithm is selected, you need to configure the parameters of multiple SADRL models and MADRL-specific parameters. The parameters of the SADRL model include DRL algorithm parameters such as neural network hyperparameters, experience pool parameters, and exploration strategy parameters. The neural network hyperparameters include the number of network layers, the number of units per layer of the network, the activation function, the learning rate, etc. The experience pool parameters include the experience pool type, maximum capacity, data dimension, reward function, etc. The exploration strategy parameters include the exploration strategy type, random probability size (or noise amplitude), etc.; MADRL-specific parameters are related to the specific MADRL algorithm used. After receiving the configuration information, each service completes the corresponding configuration and quickly applies the algorithm to build a model instance;

[0085] Step 404: Use the intelligent training tool to set simulation parameters such as simulation step length, simulation speed, and maximum simulation duration, and configure training control parameters such as the number of training games and training end conditions. After receiving the configuration information, the simulation control service completes the configuration and initializes the simulation engine.

[0086] Step 405: Use the intelligent training tool to perform control operations such as starting, pausing, continuing, and ending the intelligent training process. When the training starts, the intelligent training tool sends instructions to the simulation control service to control the start of the simulation. In each frame of simulation, the intelligent agent model obtains the situation data of the force simulation model, sends it to the intelligent reasoning service through the RPC communication middleware and obtains the calculated decision results, and passes the decision results to the force simulation model to realize the step-by-step execution of the simulation engine. The simulation data service stores the corresponding MDP data in the intelligent storage service after the step-by-step execution;

[0087] Step 406: During the training process, the intelligent optimization service repeatedly obtains batch experience data from the intelligent storage service, optimizes the neural network parameters, and pushes the updated neural network parameters to the intelligent reasoning service at a set interval. The intelligent reasoning service replaces the original model parameters with the updated parameters.

[0088] Step 407: Observe the loss curve, reward curve and other model training status on the smart training tool UI interface. If the model converges, end the training and jump to step 5; otherwise, jump to step 401.

[0089] Step 5: Deploy the trained DRL model and switch to standalone operation mode.

[0090] Specific steps:

[0091] Step 501: Use the intelligent training tool to upload the trained DRL model parameters to the intelligent reasoning service. The intelligent reasoning service solidifies the model parameters to make them unmodifiable, and closes the exploration strategy of the intelligent reasoning service to complete the deployment process of the DRL model and switch to the independent operation mode of the intelligent game training and simulation device.

[0092] Step 6: Based on the deployed DRL model, perform simulation process control, record simulation data, and observe simulation situation changes in real time. Figure 6 shown.

[0093] Specific steps:

[0094] Step 601: Use a data recording and playback tool to configure simulation data to be recorded;

[0095] Step 602: Use the simulation control tool to set simulation parameters such as simulation step length, simulation speed, and maximum simulation duration, and perform control operations such as initialization, start, pause, continue, and end of the simulation process;

[0096] Step 603: Use two-dimensional and three-dimensional situation tools to observe the changes in simulation situation in real time.

[0097] Step 7: Select simulation data and control the simulation playback process.

[0098] Specific steps:

[0099] Step 701: Use the data recording and playback tool to select the simulation data to be played back, use the simulation control tool to control the playback of the simulation deduction process, and use the two-dimensional and three-dimensional situation tool to observe the changes in the simulation situation.

[0100] Step 8: Evaluate the effectiveness of combat system weapons to determine whether the simulation purpose has been achieved, and iterate until the requirements are met.

[0101] Specific steps:

[0102] Step 801: Use the effectiveness evaluation tool to select simulation data, build an effectiveness evaluation index system, calculate the weapon effectiveness index value, and generate a comprehensive evaluation result;

[0103] Step 802: If the simulation purpose is achieved, end; otherwise, jump to the first step.

[0104] In the description of the present invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0105] The above embodiments are only descriptions of the preferred modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An operating method of an intelligent game training and simulation device for complex equipment system confrontation, characterized in that: The intelligent game training and simulation device comprises: a simulation basic environment designed with a microservice architecture, an intelligent confrontation learning training interface and an intelligent training framework, wherein the simulation basic environment is used to realize the simulation of the confrontation of the complex equipment system, and the intelligent training framework interacts with the simulation basic environment data to realize the intelligent game training of the confrontation of the complex equipment system; The intelligent training framework includes: model reasoning, experience storage and intelligent optimization, and the simulation basic environment includes: simulation engine and CGF intelligent behavior model; The microservice architecture includes: platform layer, resource layer, gateway layer, tool layer and service layer. The platform layer is used to provide underlying support for system maintenance management, container management, and orchestration; The resource layer is used to abstract various data resources and provide persistent storage and basic management operations for resources; Data resources are divided into two categories: simulation resources and intelligent resources. Simulation resources include force simulation models, combat scenario files, test design samples, simulation data records, and effectiveness evaluation projects. Intelligent resources include intelligent training tasks and DRL model parameters. The data resources at the resource layer are stored in the database in the form of microservices, supporting basic creation, reading, updating, and deletion of various resources. The intelligent agent model and the force simulation model are assembled to form an intelligent force model, which collects the situation information of the force simulation model, calls the interface of the intelligent reasoning service to obtain and execute behavioral decisions; The gateway layer is the unified entrance of the platform, routing API requests, and can perform identity authentication and resource caching; The tool layer is used to provide a human-computer interaction interface; The platform layer adopts a cloud operating system and is specifically used to implement at least one of the following functions: Virtualization, container services, service discovery, storage orchestration, automatic resource scheduling, cluster management and monitoring, log collection and storage, and service definition service deployment; The resource layer is specifically used to implement at least one of the following functions: Simulation model management, data management, scenario management, document management, data type management, map services, situation-related resource management, and intelligent resource management; The gateway layer is specifically used to implement at least one of the following functions: Authentication, load balancing, caching, and monitoring; The tool layer specifically includes at least one of the following: Simulation control tools, scenario editing tools, intelligent training tools, 2D and 3D situation tools, model management tools, test design tools, performance evaluation tools, and data recording and playback tools; The service layer includes an intelligent reasoning service for the model reasoning, an intelligent storage service for the experience storage, an intelligent optimization service for network optimization, and a simulation engine service for the simulation engine; The service layer can be centrally deployed on a single computer or distributed on multiple computers, and the service layer also includes: Scenario generation services, experimental design services, model management services, performance evaluation services, simulation control services and simulation data services for the simulation infrastructure environment; The simulation engine service includes at least one of the following: Topic management, model-driven scenario analysis, simulation control, time management and data interaction; Among them, the intelligent reasoning service, intelligent storage service and the intelligent optimization service all adopt a unified external interface, and each has multiple DRL algorithm components built in, and the simulation engine adopts a unified interface of multi-granularity models; The intelligent adversarial learning training interface is configured to receive and send message middleware to realize data interaction between the intelligent training framework and the simulation basic environment; Wherein, the message middleware includes: simulation parameters, system status, situation information and behavior decision; The intelligent adversarial learning training interface includes: A parameter configuration interface, used for receiving and sending the simulation parameters; A status monitoring interface, used to receive and send the system status; Situation acquisition interface, used to receive and send the situation information; A behavior execution interface, used to receive and send the behavior decision; The following steps are involved: Step 1: Develop force simulation models and intelligent agent models. The intelligent agent model implements the logic of obtaining the required situation data and executing decision results, and is combined with the force simulation model to form an intelligent force model, which is uploaded to the model resource library. Step 2: Develop simulation scenarios, determine combat areas, deploy force entities, set combat mission processes, configure model interaction relationships, and generate simulation scenario files; Step 3: Based on the simulation scenario file, select the test factors and test design methods, and generate the test design sample space file; Step 4: In the model training mode, create an intelligent training task and configure the situation data and decision data that need to interact with the simulation system; develop and upload a custom DRL algorithm, or directly use the built-in DRL algorithm, configure the DRL algorithm parameters, simulation parameters, and training control parameters; perform intelligent training process control and iterate until the model converges; Step 5: Deploy the trained DRL model and switch to standalone operation mode; Step 6: Based on the deployed DRL model, perform simulation process control, record simulation data, and observe simulation situation changes in real time; Step 7: Select simulation data and perform simulation playback process control; Step 8: Evaluate the effectiveness of combat system weapons to determine whether the simulation purpose has been achieved, and iterate until the requirements are met.