Intelligent simulation interaction system, method, device and medium based on reinforcement learning

Abstract

The present application provides an intelligent simulation interaction system and method, device, and medium based on reinforcement learning, which belongs to the field of reinforcement learning technology. The system includes: a task simulation module determines multiple target intelligent objects according to the task intelligent agent attribute information and the initial intelligent agent action information, and obtains the candidate environment situation data of each target intelligent object in the simulation environment; the environment interaction module compares the state of the candidate environment situation data with the task target information, so that the reinforcement learning algorithm module makes decisions on multiple target intelligent objects to obtain intelligent agent decision data; the task simulation module adjusts the intelligent agent action of the target intelligent object according to the intelligent agent decision data to obtain the adjusted environmental situation data, and compares the intelligent agent state of the adjusted environmental situation data again until the state information of the target intelligent object meets the intelligent agent target state information. The present application can improve the applicability of reinforcement learning in different task simulation interactions.

Description

Technical Field

[0001] The present application relates to the field of reinforcement learning technology, and in particular to an intelligent simulation interaction system and method, device, and medium based on reinforcement learning. Background Art

[0002] Reinforcement learning is to let the agent learn how to make decisions through interaction with the environment in order to maximize the expected reward. Reinforcement learning is often used in tasks that require continuous decision-making and feedback, such as drone confrontation, autonomous driving, robot control, etc. In reinforcement learning, the agent learns the optimal decision-making strategy through the process of interacting with the environment to achieve specific goals. Since there is a lot of trial and error behavior in the interaction between the agent and the real environment, the training cost in the real environment is relatively high. Therefore, a simulation application system is usually used to provide the agent with a safe and controllable training environment to help the agent accelerate its learning process.

[0003] However, since the development of reinforcement learning algorithms is not synchronized with the development of simulation application systems, reinforcement learning algorithms cannot directly interact effectively with existing simulation software systems, resulting in a limited training process, which affects its application efficiency in specific task areas. In order to solve this problem, related technologies usually customize the development of corresponding simulation application systems for reinforcement learning in specific fields to adapt to the simulation environment required for different tasks. However, this method greatly increases the complexity of simulation application software development, and consumes a lot of resources in simulation application software development. Therefore, how to provide an intelligent simulation interaction system based on reinforcement learning that can adapt to the simulation environment required for different tasks and avoid large resource consumption in the development of simulation application systems, thereby improving the application efficiency of reinforcement learning in different task areas, has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an intelligent simulation interaction system and method, device, and medium based on reinforcement learning, which can adapt to the simulation environment required by different tasks, avoid large resource consumption in the development of simulation application software, and improve the application efficiency of reinforcement learning in different task fields, thereby improving the applicability of reinforcement learning in simulation interaction in different task fields.

[0005] To achieve the above-mentioned purpose, the first aspect of the embodiment of the present application proposes an intelligent simulation interaction system based on reinforcement learning, the system includes an interactive client, a task simulation module and a target decision platform, the target decision platform includes a reinforcement learning algorithm module and an environment interaction module, the reinforcement learning algorithm module is communicatively connected with the interactive client, and the environment interaction module is communicatively connected with the interactive client and the task simulation module respectively;

[0006] The interactive client is used to send the algorithm configuration information of the target task to the reinforcement learning algorithm module, and send the environment configuration information of the target task to the environment interaction module; wherein the environment configuration information includes task agent attribute information, initial agent action information and task target information, and the task target information is used to indicate the agent target state information when the target task is completed;

[0007] The task simulation module is used to perform physical mapping in the simulation environment according to the received task agent attribute information and initial agent action information, and determine multiple target agent objects of the target task in the simulation environment; the task simulation module is also used to obtain information after each target agent object acts in the simulation environment in real time, obtain candidate environment situation data, and send the candidate environment situation data to the environment interaction module;

[0008] The environment interaction module is used to compare the agent state according to the candidate environment situation data and the task target information to obtain a state comparison result; if the state comparison result indicates that the state information of the multiple target agent objects corresponding to the candidate environment situation data does not conform to the agent target state information, the candidate environment situation data is sent to the reinforcement learning algorithm module;

[0009] The reinforcement learning algorithm module is used to obtain the environment configuration information and the candidate environment situation data from the environment interaction module; the reinforcement learning algorithm module is also used to make action decisions for the multiple target intelligent agents according to the task target information and the candidate environment situation data, obtain intelligent agent decision data, and send the intelligent agent decision data to the environment interaction module;

[0010] The task simulation module is also used to adjust the agent actions of multiple target agent entities in the simulation environment according to the agent decision data to obtain adjusted environmental situation data, and send the adjusted environmental situation data to the environment interaction module to perform agent state comparison again to update the state comparison result until the state comparison result indicates that the state information of the multiple target agent entities corresponding to the adjusted environmental situation data is consistent with the agent target state information.

[0011] In some embodiments, the target decision platform includes a semantic interaction layer, and the semantic interaction layer is respectively communicatively connected with the interactive client, the reinforcement learning algorithm module and the environment interaction module;

[0012] The semantic interaction layer is used to receive the algorithm configuration information and the environment configuration information from the interactive client, wherein the algorithm configuration information includes the scene configuration information of the reinforcement learning scene, and the scene configuration information is used to guide the reinforcement learning algorithm module to make action decisions for the plurality of target intelligent entities according to the task target information and the environmental situation data received in real time;

[0013] The semantic interaction layer is also used to receive a target custom function related to the task target information from the interactive client when the candidate environment situation data does not contain task reward sub-data, and send the target custom function to the reinforcement learning algorithm module, the target custom function is used to calculate the task reward according to the candidate environment situation data to obtain the task reward sub-data, and the task reward sub-data is used to generate the agent decision data.

[0014] In some embodiments, the target decision platform includes an environment interaction layer, the environment interaction layer is communicatively connected with the environment interaction module and the task simulation module, and the environment interaction layer includes a first connection interface, an initialization interface, and an interaction sending interface;

[0015] The first connection interface is used to connect to the task simulation module;

[0016] The initialization interface is used to initialize the task simulation module when the task simulation module completes configuration input; the initialization interface is also used to restore the task simulation module to the initialization state at startup after determining that the state information of the plurality of target intelligent agent objects corresponding to the adjusted environmental situation data conforms to the intelligent agent target state information;

[0017] The interactive sending interface is used to call the interactive class interface of the task simulation module to send the agent decision data obtained by the environment interaction module from the reinforcement learning algorithm module to the task simulation module; the interactive sending interface is also used to send the adjusted environmental situation data obtained after adjustment of the task simulation module to the environment interaction module.

[0018] In some embodiments, the environment interaction layer further includes a log management interface and a cross-language calling interface;

[0019] The log management interface is used to record all data interacting between the task simulation module and the environment interaction module;

[0020] The cross-language calling interface is used to implement interface calling and data exchange between different programming languages ​​connected to the target decision platform.

[0021] In some embodiments, the target decision platform further includes an algorithm interaction layer, the algorithm interaction layer is communicatively connected with the reinforcement learning algorithm module and the environment interaction module, and the algorithm interaction layer includes a second connection interface, a reset interface, and an acquisition interface;

[0022] The second connection interface is used to establish a communication connection between the algorithm interaction layer and the environment interaction layer before the reinforcement learning training is started; the second connection interface is also used to call the first connection interface;

[0023] The reset interface is used to call the first connection interface and reset the task simulation module after determining that the state information of the plurality of target intelligent agent objects corresponding to the adjusted environmental situation data conforms to the intelligent agent target state information, so as to perform the next round of reinforcement learning training;

[0024] The acquisition interface is used to obtain the agent decision data output by the reinforcement learning algorithm module, and the acquisition interface is also used to obtain the candidate environment situation data output by the environment simulation module from the environment interaction module.

[0025] In some embodiments, the environment interaction module is used to compare the agent state according to the candidate environment situation data and the task target information to obtain the state comparison result, specifically including:

[0026] The environment interaction module is used to receive the candidate environment situation data from the task simulation module, and perform data format conversion on the candidate environment situation data to obtain converted environment situation data;

[0027] The environment interaction module is also used to compare the state of the intelligent body according to the converted environment situation data and the task target information to obtain the state comparison result.

[0028] In some embodiments, after updating the state comparison result until the state comparison result indicates that the state information of the plurality of target intelligent agent objects corresponding to the adjusted environmental situation data conforms to the intelligent agent target state information, the environmental interaction module is further used to extract reward data from all environmental situation data received by the environmental interaction module to obtain a reward data sequence;

[0029] The environment interaction module is also used to generate a reward process curve according to the reward data sequence, and send the reward process curve to the interactive client.

[0030] To achieve the above-mentioned purpose, the second aspect of the embodiment of the present application proposes an intelligent simulation interaction method based on reinforcement learning, which is applied to the target decision platform in the intelligent simulation interaction system based on reinforcement learning proposed in the first aspect of the embodiment of the present application, and the method includes:

[0031] Receive algorithm configuration information and environment configuration information of the target task sent by the interactive client; wherein the environment configuration information includes task agent attribute information, initial agent action information and task target information, and the task target information is used to indicate the agent target state information when completing the target task;

[0032] Receiving candidate environment situation data sent by the task simulation module according to the task agent attribute information and the initial agent action information, wherein the candidate environment situation data is used to represent information after each target agent object related to the target task acts in real time in the simulation environment;

[0033] Comparing the state of the agent with the candidate environment situation data and the task target information to obtain a state comparison result;

[0034] If the state comparison result indicates that the state information of the plurality of target intelligent agent objects corresponding to the candidate environmental situation data does not conform to the intelligent agent target state information, action decisions are made for the plurality of target intelligent agent objects according to the task target information and the candidate environmental situation data to obtain intelligent agent decision data, and the intelligent agent decision data is sent to the task simulation module;

[0035] Receiving the adjusted environment situation data obtained by the task simulation module after adjusting the agent actions of the plurality of target agent entities in the simulation environment according to the agent decision data;

[0036] The agent state comparison is performed again based on the adjusted environmental situation data and the task target information to update the state comparison result, until the state comparison result indicates that the state information of the multiple target agent objects corresponding to the adjusted environmental situation data is consistent with the agent target state information.

[0037] To achieve the above objective, a third aspect of an embodiment of the present application provides a computer device, including:

[0038] at least one memory;

[0039] at least one processor;

[0040] at least one computer program;

[0041] The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement the intelligent simulation interaction method based on reinforcement learning described in the second aspect above.

[0042] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program, and the computer program is used to enable a computer to execute the intelligent simulation interaction method based on reinforcement learning described in the second aspect above.

[0043] The embodiment of the present application proposes an intelligent simulation interaction system and method, device, and medium based on reinforcement learning. The system includes an interactive client, a task simulation module, and a target decision platform. The target decision platform includes a reinforcement learning algorithm module and an environment interaction module. The reinforcement learning algorithm module is connected to the interactive client in communication, and the environment interaction module is connected to the interactive client and the task simulation module in communication. The present application can connect the environment interaction module with the external task simulation module through the environment interaction module and the reinforcement learning algorithm module in the set target decision platform, so as to realize the simulation interaction under the simulation environment of different tasks. Specifically, the interactive client is used to send the algorithm configuration information of the target task to the reinforcement learning algorithm module, and send the environment configuration information of the target task to the environment interaction module. Among them, the environment configuration information includes the task agent attribute information, the initial agent action information, and the task target information, and the task target information is used to indicate the agent target state information when the target task is completed. The task simulation module is used to perform physical mapping in the simulation environment according to the received task agent attribute information and initial agent action information, determine the target task in the simulation environment of multiple target agent objects, and the task simulation module is also used to obtain the information of each target agent object after real-time action in the simulation environment, obtain candidate environment situation data, and send the candidate environment situation data to the environment interaction module. The environment interaction module is used to compare the agent state according to the candidate environment situation data and the task target information to obtain the state comparison result. If the state comparison result indicates that the state information of the multiple target agent objects corresponding to the candidate environment situation data does not conform to the agent target state information, the candidate environment situation data is sent to the reinforcement learning algorithm module. At this time, the reinforcement learning algorithm module is used to obtain the environment configuration information and the candidate environment situation data from the environment interaction module; the reinforcement learning algorithm module is also used to make action decisions for the multiple target agent objects according to the task target information and the candidate environment situation data, obtain the agent decision data, and send the agent decision data to the environment interaction module. Then, the task simulation module is also used to adjust the agent actions of multiple target agent objects in the simulation environment according to the agent decision data, obtain the adjusted environmental situation data, and send the adjusted environmental situation data to the environment interaction module to perform agent state comparison again to update the state comparison result, until the state comparison result indicates that the state information of multiple target agent objects corresponding to the adjusted environmental situation data is consistent with the agent target state information. Therefore, the target decision platform is equivalent to an intermediate device, which can well implement the reinforcement learning training of different task simulation modules, that is, compared with the related technology that can only develop simulation application systems corresponding to each task through different customization, this application can adapt to the simulation environment required by different tasks, avoid large resource consumption in the development of simulation application software, thereby improving the application efficiency and applicability of reinforcement learning in different task fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of a module structure of an intelligent simulation interaction system based on reinforcement learning provided in an embodiment of the present application;

[0045] Figure 2 It is a structural diagram of the internal layers of the target decision-making platform provided in the embodiment of the present application;

[0046] Figure 3 It is a schematic diagram of an interface flow of the internal level of the target decision-making platform provided in the embodiment of the present application;

[0047] Figure 4 It is a step flow chart of the intelligent simulation interaction method based on reinforcement learning provided in an embodiment of the present application;

[0048] Figure 5 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0052] First, some nouns involved in this application are analyzed:

[0053] Simulation interaction: refers to the process of simulating (emulating) in computer software or system and interacting with users or other systems. In simulation interaction, the system will simulate a certain actual scene or process, and users can observe the simulation results, adjust parameters or control through interactive operations, thereby achieving influence and control over the simulation process.

[0054] Reinforcement Learning: It is a machine learning method that aims to enable an agent to learn how to obtain the maximum cumulative reward in a task through interaction with the environment. In reinforcement learning, the agent learns how to make the best decision to achieve the given goal by trying different actions and observing the feedback (reward signal) from the environment.

[0055] Remote Procedure Call (RPC): is a communication framework for implementing remote procedure calls. RPC allows communication between different computers or processes, so that they can call functions in remote computers or processes just like calling local functions. The RPC framework usually consists of two parts: the client and the server. The client initiates a remote call request, the server receives the request and executes the corresponding remote procedure, and then returns the result to the client. The RPC framework can simplify communication and data transmission in distributed systems, making it easier for different parts of the system to work together. Common RPC frameworks include gRPC, Apache Thrift, Dubbo, etc., which provide a variety of functions and features to meet the needs of different scenarios.

[0056] Application Programming Interface (API): is a set of interface specifications that define how software applications communicate and interact with other software, hardware, or systems. API allows different software systems to share data, functions, and services, thereby achieving integration and interoperability between systems. API can be regarded as a bridge between software systems, which defines how to communicate and interact through predefined interfaces (including methods, parameters, data formats, etc.).

[0057] JSON (JavaScript Object Notation): is a data exchange format, commonly used to transfer data between different systems. It is a lightweight text data format that is easy to read and write. The JSON format is often used to represent structured data such as objects, arrays, strings, etc.

[0058] File format: refers to the organization structure and storage method of files. File formats can be of various types, such as text files (.txt), image files (jpg, png), audio files (.mp3), video files (.mp4), etc. Each file format has its own specific storage structure and specifications for storing the corresponding type of data.

[0059] Reinforcement learning is to let the agent learn how to make decisions through interaction with the environment in order to maximize the expected reward. Reinforcement learning is often used in tasks that require continuous decision-making and feedback, such as drone confrontation, autonomous driving, robot control, etc. In reinforcement learning, the agent learns the optimal decision-making strategy through the process of interacting with the environment to achieve specific goals. Since there is a lot of trial and error behavior in the interaction between the agent and the real environment, the training cost in the real environment is relatively high. Therefore, a simulation application system is usually used to provide the agent with a safe and controllable training environment to help the agent accelerate its learning process.

[0060] However, since the development of reinforcement learning algorithms is not synchronized with the development of simulation application systems, reinforcement learning algorithms cannot directly interact effectively with existing simulation software systems, resulting in limited training processes, which in turn affects their application efficiency in specific task areas. In order to solve this problem, related technologies usually customize the development of corresponding simulation application systems for reinforcement learning in specific fields to adapt to the simulation environment required by different tasks. However, this method that requires customized development to adapt to different simulation environments greatly increases the complexity of simulation application software development, and consumes a lot of resources in simulation application software development. Therefore, how to provide an intelligent simulation interaction system based on reinforcement learning that can adapt to the simulation environment required by different tasks and avoid large resource consumption in the development of simulation application systems, thereby improving the application efficiency of reinforcement learning in different task areas, has become a technical problem that needs to be solved urgently.

[0061] Based on this, the embodiments of the present application provide an intelligent simulation interaction system and method, device, and medium based on reinforcement learning, which can adapt to the simulation environment required by different tasks, avoid large resource consumption in the development of simulation application software, and thus improve the application efficiency of reinforcement learning in different task fields.

[0062] The intelligent simulation interaction method based on reinforcement learning provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms; the software can be an application that implements the intelligent simulation interaction method based on reinforcement learning, etc., but is not limited to the above forms.

[0063] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network personal computers (PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0064] See also Figure 1 , Figure 1 is a schematic diagram of a module interaction of an intelligent simulation interaction system based on reinforcement learning provided in an embodiment of the present application. In some embodiments of the present application, Figure 1 The intelligent simulation interaction system 100 in the embodiment may specifically include an interactive client 110, a task simulation module 120 and a target decision platform 130, wherein the target decision platform 130 includes a reinforcement learning algorithm module 131 and an environment interaction module 132, wherein the reinforcement learning algorithm module 131 is connected to the interactive client 110 in communication, and the environment interaction module is connected to the interactive client and the task simulation module in communication, as described below. Figure 1 The function of each module is introduced in detail.

[0065] The interactive client 110 is used to send the algorithm configuration information of the target task to the reinforcement learning algorithm module 131 , and send the environment configuration information of the target task to the environment interaction module 132 .

[0066] It should be noted that the interactive client 110 is a program or system that allows the user to interact with the reinforcement learning algorithm module and the environment interaction module. The user can input the relevant configuration information of the target task through the interactive client 110 and receive the output results from the two modules, namely, the algorithm output of the reinforcement learning algorithm module and the interactive analysis result output by the environment interaction module.

[0067] It should be noted that the reinforcement learning algorithm module 131 is a component for processing reinforcement learning algorithms, which trains the agent according to the algorithm configuration information of the received target task and the data input from the environment interaction module 132 during the training process. Therefore, the reinforcement learning algorithm module 131 forms a general reinforcement learning simulation environment interaction framework, i.e., a target decision platform, through the connection with the environment interaction module 132, which is conducive to the integrated development and management of reinforcement learning algorithms to learn how to complete tasks in order to achieve the goal of maximizing cumulative rewards.

[0068] It should be noted that the environment interaction module 132 is a component that connects the external simulation environment and the reinforcement learning algorithm. That is, the environment interaction module 132 can send the real-time changing situation data received from the task simulation module 120 to the reinforcement learning algorithm module 131, and can also return the agent decision data after the reinforcement learning algorithm module 131 processes and analyzes the real-time changing situation data to the task simulation module 120. In this way, through the design of the reinforcement learning algorithm module 131-environment interaction module 132-task simulation module 120, the interaction between different simulation services and the reinforcement learning algorithm is realized, thereby realizing pluggable simulation environment access.

[0069] It should be noted that the target task is used to characterize the task scenario for simulation, that is, different target tasks can be determined according to different task objects. For example, when the task object served is an aircraft radar detection mechanism, the corresponding simulation environment and target task are related to the radar business scenario; when the task object served is related to a drone, the corresponding simulation environment and target task are related to the drone confrontation exercise scenario. In this way, the algorithm configuration information of the target task is used to characterize a set of parameter settings for the reinforcement learning algorithm, and the algorithm configuration information can be used to tell the reinforcement learning algorithm module 131 how to train the agent to complete a specific task. This information may include learning rate, discount factor, exploration strategy, etc.

[0070] It should be noted that the environment configuration information of the target task is a set of environment parameters and settings that describe the target task, which is used to tell the environment interaction module 132 how to create a virtual environment suitable for training the agent. The environment configuration information includes task agent attribute information, initial agent action information and task target information, and the task target information is used to indicate the agent target state information when the target task is completed. Among them, the task agent attribute information is used to characterize the number of agents and agent types required for the target task reinforcement learning scenario and other attribute information related to the agent. The initial agent action information is used to characterize the observation space and action space information of the initial state of the agent. The task target information is used to characterize the state information when the target task is completed. For example, if the target task is an air combat patrol in a drone confrontation exercise scenario, and the task is set to include the A and B parties, each party is configured with 5 drones and 2 radar bases, and the battlefield attribute configuration is set, then the task agent attribute information contains 15 agents, and includes 10 drone-type agents, 4 radar base-type agents and one battlefield-type agent. The initial agent action information includes the location set by party A as "longitude A1, latitude A2, heading A3, altitude, action status", etc. In addition, the mission target information at this time may be "if one party reaches the radar base of the other party, the confrontation exercise ends".

[0071] Among them, the task agent attribute information describes the attributes of the agent when performing the task, such as the size, shape, speed, perception range, etc. of the agent. These attribute information helps the reinforcement learning algorithm module to accurately predict the behavior of each agent in the target task. The initial agent action information describes the initial state of each agent at the beginning of the task, such as the position, speed, direction, etc. of the agent. This information helps the reinforcement learning algorithm module to set a suitable initial state for the agent so that the behavior of the agent can be accurately simulated at the beginning of the task. The task target information describes the agent target state information required to complete the task, such as the target position that the agent needs to reach, the number of tasks that need to be completed, etc. This information helps the reinforcement learning algorithm module to set a clear goal for the agent so that the behavior of the agent can be accurately guided during the training process. Through the algorithm configuration information and environment configuration information input by the interactive client 110, the configuration information related to the target task can be accurately obtained, thereby providing an accurate model and environment for the agent, so that the behavior of the agent can be accurately predicted and controlled during the training process.

[0072] It should be noted that the reinforcement learning algorithm module 131 is mainly used to develop and integrate reinforcement learning algorithms and is the main body of reinforcement learning training. Figure 1The most critical part of the framework shown plays the role of a bridge for the interaction between the external simulation environment and the reinforcement learning algorithm, and its communication mechanism with the outside mainly adopts the remote procedure call method. Furthermore, the environment interaction module 132 can adopt the RPC framework or other frameworks, which are not specifically limited here. Among them, the RPC framework can specifically include Dubbo, Thrift, gRPC and Finagle, etc. The actual selection can be combined with the needs of specific scenarios and technical capabilities. Therefore, the interactive client 110 can be integrated into a unified target decision platform 130 for realizing intelligent decision-making development through the reinforcement learning algorithm module 131 and the environment interaction module 132 (also referred to as the environment engine), and the corresponding algorithm configuration information and environment configuration information are input through the interactive client 110, which is conducive to providing a more convenient simulation training environment based on reinforcement learning.

[0073] In some embodiments, the task simulation module 120 is used to perform physical mapping in the simulation environment based on the received task agent attribute information and initial agent action information, and determine multiple target agent entities for the target task in the simulation environment; the task simulation module 120 is also used to obtain information after each target agent entity acts in the simulation environment in real time, obtain candidate environment situation data, and send the candidate environment situation data to the environment interaction module 132.

[0074] Among them, the task simulation module 120 is an external structure for building a simulation environment. Physical mapping refers to the process of mapping multiple actual target agent objects in the simulation environment based on the task agent attribute information and the initial agent action information. This process involves creating an agent model based on the task agent attribute information, and setting the initial state of the agent based on the initial agent action information. It is understandable that the task simulation module 120 is a simulation environment provided by a third party and is the real provider of simulation services. The task simulation module 120 is a replaceable part and can be flexibly replaced according to different target tasks. Therefore, the target decision platform 130 integrated with the environment interaction module 132 can support the reinforcement learning training needs of multiple fields and scenarios, and has universality. In addition, the task simulation module 120 provides an external call interface, which can allow secondary development of it.

[0075] It should be noted that the assumption configuration input to the task simulation module 120 is the configuration information of the scene corresponding to the simulated target task. For example, if the target task is a scene of drone confrontation, then the input assumption configuration is the simulation environment configuration information related to the drone confrontation exercise scene. Among them, the assumption configuration and the above-mentioned environmental configuration information input to the environment interaction module 132 are the same, but there are differences in the format of the information to adapt to the information format requirements inside the environment interaction module 132 and the task simulation module 120 respectively. For example, the input configurations in the task simulation module 120 are all in JSON format, and the environmental configuration information input to the environment interaction module 132 by the interactive client 110 can be transmitted in the form of file format, then the environment interaction module 132 can parse the received file to reversely determine the simulation object.

[0076] It should be noted that the candidate environment situation data is a set of data collected in real time in the simulation environment, which is used to describe the state of each target agent after real-time action. The candidate environment situation data may include the observation value of each agent on the simulation environment information and the observation value of the agent on its own state, that is, it may include the position, speed, direction and other information of the target agent corresponding to the agent at the current moment, and its specific value is output by the external task simulation module.

[0077] Furthermore, the task simulation module 120 sends the candidate environment situation data to the environment interaction module 132 in real time so that the data can be continuously used to guide the behavior of each target intelligent agent during the training process.

[0078] In some embodiments, Figure 2 As shown, the target decision platform 130 includes a semantic interaction layer, which is respectively communicated with the interactive client, the reinforcement learning algorithm module and the environment interaction module.

[0079] The semantic interaction layer is used to receive algorithm configuration information and environment configuration information from the interactive client. In other words, the semantic interaction layer is used to connect the interactive client and the target decision platform.

[0080] It should be noted that the algorithm configuration information includes scenario configuration information of the reinforcement learning scenario, and the scenario configuration information is used to guide the reinforcement learning algorithm module to make action decisions for multiple target intelligent entities based on task objective information and real-time received environmental situation data.

[0081] Among them, the scenario configuration information is used to characterize the effective environment configuration required for the reinforcement learning scenario, and also represents the configuration information of the external simulation environment, which mainly includes the number of agents, agent types, and the observation space information and action space information of the corresponding type of agents. Therefore, the input of the scenario configuration information is determined by the external task simulation module and is consistent with it. The observation space information includes the target agent's observation value of the environment information and the agent's observation value of its own state, and its specific value is output by the external simulation environment; the action space information is the decision action space of the target agent, and its specific value is output by the reinforcement learning algorithm module.

[0082] It should be noted that the environment configuration information also includes environment connection information and environment API information. Among them, the environment connection information is used to characterize the Internet Protocol (IP) and port information that provide simulation services for the external task simulation module. The environment API information mainly includes three types of API information, namely environment initialization class, state acquisition class and environment interaction class. The state acquisition class includes environment state acquisition, agent state acquisition interface and target completion state acquisition, etc. The environment interaction class is mainly an agent decision calling interface, which is used to obtain the agent decision data output by the reinforcement learning algorithm module from the environment interaction module of the target decision platform. The environment API information specifically also includes the API name, the name and type examples of the input and output parameters, and its purpose is to facilitate the serialization and deserialization parsing of the interaction information by the environment interaction layer.

[0083] Among them, the semantic interaction layer is also used to receive a target custom function related to the task target information from the interactive client when the candidate environment situation data does not contain the task reward sub-data, and send the target custom function to the reinforcement learning algorithm module. The target custom function is used to calculate the task reward according to the candidate environment situation data to obtain the task reward sub-data, and the task reward sub-data is used to generate the intelligent agent decision data.

[0084] It should be noted that the target custom function is used to characterize the reward function that guides the agent to complete the target task. For example, the target task is a drone confrontation exercise task, and the task target information can be "If one party reaches the radar base of the other party, the confrontation exercise ends." If the distance that one party's drone moves toward the radar base of the other party is close, then the reinforcement learning algorithm module will give the target agent corresponding to the moving drone a positive reward; if the distance that the target agent corresponding to the moving drone moves away from the radar base of the other party becomes farther, then the reinforcement learning algorithm module will deduct the corresponding reward from the target agent corresponding to the moving drone. For example, the target custom function defines a reward value of 25 for cooperative execution of the task and a reward value of 20 for successful shooting. Other reward types can also be defined according to the needs of the scene. Therefore, in the process of confrontation reinforcement learning training, the reinforcement learning algorithm module will guide the target agent to gradually approach the task target through positive rewards. It should be noted that when the target reward function or custom reward function is not provided in the assumption configuration of the external task simulation module, the target custom function can be set in the interactive client. Among them, the interactive client can automatically integrate the acquisition interface (also called step interface) of the algorithm interaction layer by providing a standardized Python code editing interface, and the standardized structure refers to fixed function names, input parameters, and output parameter types.

[0085] Among them, the task reward sub-data is used to characterize the reward value of the target intelligent agent determined based on the current candidate environment situation data, and then combined with other observation values, action state values, etc. related to the target intelligent agent to make the next action decision to generate the intelligent agent decision data.

[0086] In some embodiments, the environment interaction module 132 is used to compare the agent state according to the candidate environment situation data and the task target information to obtain a state comparison result; if the state comparison result indicates that the state information of multiple target agent objects corresponding to the candidate environment situation data does not conform to the agent target state information, the candidate environment situation data is sent to the reinforcement learning algorithm module.

[0087] It should be noted that when the environment interaction module receives the candidate environment situation data sent by the task simulation module, it will first compare the candidate environment situation data with the task target information to determine whether the state information of the multiple target intelligent agent objects corresponding to the candidate environment situation data is consistent with the intelligent agent target state information, that is, whether the multiple target intelligent agent objects after the current decision adjustment have completed the target task in the simulation environment. If the state comparison result indicates that the state information of the multiple target intelligent agent objects corresponding to the candidate environment situation data does not meet the intelligent agent target state information, the candidate environment situation data is sent to the reinforcement learning algorithm module, and the corresponding intelligent agent decision data is continued to be issued to the multiple target intelligent agent objects in the simulation environment according to the reinforcement learning algorithm model to guide whether the multiple target intelligent agent objects have completed the target task in the simulation environment.

[0088] In other embodiments, if the state comparison result indicates that the state information of multiple target intelligent agent objects corresponding to the candidate environmental situation data is consistent with the intelligent agent target state information, the environmental interaction module is also used to extract reward data from all environmental situation data received by the environmental interaction module to obtain a reward data sequence. Then, a reward process curve is generated based on the reward data sequence, and the reward process curve is sent to the interactive client. In other words, if multiple target intelligent agent objects complete the target task in the simulation environment, the environmental interaction module will obtain the target achievement progress and reward process curve of the target task for display on the interactive client.

[0089] In some specific embodiments, the environment interaction module is used to compare the agent state according to the candidate environment situation data and the task target information to obtain the state comparison result, which specifically includes:

[0090] The environment interaction module is used to receive candidate environment situation data from the task simulation module, and convert the candidate environment situation data into a data format to obtain converted environment situation data;

[0091] The environment interaction module is also used to compare the state of the intelligent agent according to the converted environment situation data and the task target information to obtain the state comparison result.

[0092] Among them, the environment interaction module first receives candidate environment situation data from the task simulation module. These data may include current situation information about multiple target agents in the simulation environment, such as position, speed, attributes, etc. Since there are differences in the data transmission formats between the environment interaction module and the task simulation module, after receiving the candidate environment situation data from the task simulation module, the environment interaction module can convert the data format of the candidate environment situation data to obtain the converted environment situation data. At this time, the converted environment situation data is used to characterize the data format that can be processed by the reinforcement learning algorithm module.

[0093] In other embodiments, the data format of the candidate environmental situation data may be converted in the task simulation module to obtain converted environmental situation data, and then the converted environmental situation data may be sent to the environmental interaction module.

[0094] Furthermore, the converted environmental situation data is compared with the task target information to determine whether the target agent is in the correct position, state or behavior to complete the target task. The state comparison result can also indicate whether the target agent needs to adjust its behavior or decision to better adapt to the current environment and task goals.

[0095] In some embodiments, the reinforcement learning algorithm module 131 is used to obtain environmental configuration information and candidate environmental situation data from the environmental interaction module; the reinforcement learning algorithm module is also used to make action decisions for multiple target intelligent agents based on task objective information and candidate environmental situation data, obtain intelligent agent decision data, and send the intelligent agent decision data to the environmental interaction module.

[0096] Among them, the reinforcement learning algorithm module: This is a module used to train the intelligent agent according to the received target task algorithm configuration information. The reinforcement learning algorithm module interacts with the environment interaction module to learn how to complete the task in order to achieve the task goal of maximizing the cumulative reward. The task goal information is used to guide the behavior of the intelligent agent during the training process so that it can reach the expected target state. Action decision-making is a process used to use the reinforcement learning algorithm based on the task goal information and the candidate environment situation data to determine the action strategy of multiple target intelligent agents to achieve the expected target state in a given environment.

[0097] The agent decision data is used to characterize the decision-making behavior of the target agent when performing the task. The agent decision data may include information such as the agent's action strategy, execution time, and reward signal, that is, it may include information such as the specific actions, behavior strategies, or path planning that the target agent needs to perform. Therefore, based on the received task target information and environmental situation data, the reinforcement learning algorithm module makes action decisions for multiple target agent objects, that is, selects appropriate actions or strategies to enable multiple target agent objects to move toward the task goal in the current simulation environment.

[0098] During the training process, the reinforcement learning algorithm module can determine the action strategies of multiple target intelligent agents based on the mission target information and candidate environment situation data, and collect intelligent agent decision data during the training process. This data can be used to evaluate the performance of the intelligent agent and further optimize the reinforcement learning algorithm to improve the performance of the intelligent agent.

[0099] In some specific embodiments, if the target task is a UAV confrontation exercise task, the confrontation scene of the external simulation environment can be restored in the environment interaction module. And through the real-time deduction and interaction of the external simulation environment corresponding to the task simulation module, the environment interaction module can follow the deduction situation of the external simulation environment, interact with the reinforcement learning algorithm module at the same time, and input the decision results (i.e., the agent decision data) to the external simulation environment, and update the environment situation data, i.e., including information such as object attributes.

[0100] In some embodiments, the task simulation module 120 is also used to adjust the agent actions of multiple target agent objects in the simulation environment according to the agent decision data, obtain adjusted environmental situation data, and send the adjusted environmental situation data to the environment interaction module for another agent state comparison to update the state comparison result until the state comparison result indicates that the state information of the multiple target agent objects corresponding to the adjusted environmental situation data is consistent with the agent target state information.

[0101] Among them, when the task simulation module receives the agent decision data sent by the environment interaction module, it will adjust the agent actions of multiple target agent objects in the simulation environment according to the agent decision data, and collect the observation information and action space information of each target agent object after readjustment again to obtain the adjusted environment situation data. Then, the adjusted environment situation data is sent to the environment interaction module for agent state comparison again to update the state comparison result. If the updated state comparison result still indicates that the state information of multiple target agent objects corresponding to the adjusted environment situation data does not conform to the agent target state information, the above steps are repeated until the state comparison result indicates that the state information of multiple target agent objects corresponding to the adjusted environment situation data conforms to the agent target state information, that is, a reinforcement learning training of the target task is completed.

[0102] In some embodiments, Figure 2 As shown, the target decision platform of the present application includes an environment interaction layer, which is connected to the environment interaction module and the task simulation module, that is, the environment interaction layer can obtain information from the semantic interaction layer and the algorithm interaction layer, and complete the message transmission with the external simulation environment. The environment interaction layer may include three external interfaces: a first connection interface, an initialization interface, and an interaction sending interface.

[0103] The first connection interface is used to connect to the task simulation module.

[0104] The first connection interface can be recorded as connect. Figure 3 As shown, the semantic interaction layer sends the environment connection information to the environment interaction layer through the first connection interface.

[0105] The initialization interface is used to initialize the task simulation module when the task simulation module completes the configuration input; the initialization interface is also used to restore the task simulation module to the initialization state at startup after determining that the state information of multiple target intelligent agent objects corresponding to the adjusted environmental situation data conforms to the intelligent agent target state information.

[0106] Among them, the initialization interface can be recorded as init. The semantic interaction layer can call the initialization interface corresponding to the external simulation environment to complete the initialization of the simulation environment. Specifically, the semantic interaction layer can initialize the task simulation module according to the environment API information by calling the initialization interface when the task simulation module completes the configuration input. In addition, the semantic interaction layer can also restore the task simulation module to the initialization state at the time of startup after determining that the state information of multiple target intelligent agent objects corresponding to the adjusted environmental situation data is consistent with the intelligent agent target state information by calling the initialization interface according to the scene configuration information, that is, after a certain round of training of the reinforcement learning algorithm is completed, the simulation environment is restored to the initialization state at the time of startup.

[0107] The interactive sending interface is used to call the interactive class interface of the task simulation module to send the intelligent agent decision data obtained by the environment interaction module from the reinforcement learning algorithm module to the task simulation module; the interactive sending interface is also used to send the adjusted environmental situation data obtained after the task simulation module is adjusted to the environment interaction module.

[0108] The interactive sending interface can be recorded as send. The interactive sending interface can call the interactive interface of the task simulation module. At this time, the input information is the agent decision data corresponding to the target agent, and the result returns the adjusted environmental situation data of the corresponding target agent, including action space data, environmental observation data, and task completion information.

[0109] In some embodiments, the environment interaction layer also includes a log management interface and a cross-language calling interface; wherein the log management interface is used to record all data of the interaction between the task simulation module and the environment interaction module; the cross-language calling interface is used to realize interface calling and data exchange between different programming languages ​​connected to the target decision platform.

[0110] In other embodiments, the environment interaction layer also includes functional interfaces such as connection disconnection, abnormal data acquisition, and data storage, which are not specifically limited here.

[0111] In some embodiments, the target decision platform further includes an algorithm interaction layer, which is connected to the reinforcement learning algorithm module and the environment interaction module, that is, the algorithm interaction layer obtains information by calling the environment interaction layer interface and completes the interaction with the reinforcement learning algorithm module. Among them, the algorithm interaction layer may include three external interfaces, namely, a second connection interface, a reset interface, and an acquisition interface.

[0112] The second connection interface is used to establish a communication connection between the algorithm interaction layer and the environment interaction layer before the reinforcement learning training is started; the second connection interface is also used to call the first connection interface. The second connection interface can be recorded as make_env.

[0113] The reset interface is used to call the first connection interface, and after determining that the state information of multiple target intelligent agent objects corresponding to the adjusted environmental situation data conforms to the intelligent agent target state information, the task simulation module is reset to carry out the next round of reinforcement learning training.

[0114] The reset interface can be recorded as reset. Figure 3 As shown, the reset interface can be connected to the initialization interface of the environment interaction layer to complete the reset of the simulation environment and facilitate entering the next round of training.

[0115] The acquisition interface is used to obtain the agent decision data output by the reinforcement learning algorithm module. The acquisition interface is also used to obtain the candidate environment situation data output by the environment simulation module from the environment interaction module.

[0116] The acquisition interface can be recorded as step. The algorithm interaction layer can pass the agent decision data output by the reinforcement learning algorithm model to this interface. In addition, the acquisition interface can call the interactive sending interface to obtain information such as algorithm configuration information and candidate environment situation data of the agent from the environment interaction layer.

[0117] In the above embodiments, the embodiments of the present application realize the interaction between the simulation service corresponding to different tasks and the reinforcement learning algorithm by implementing a three-layer design in the environment interaction module, namely, including a semantic interaction layer, an environment interaction layer and an algorithm interaction layer, that is, realizing pluggable simulation environment access. Specifically, the semantic interaction layer provides information input for the environment interaction layer and the algorithm interaction layer. The algorithm interaction layer provides an algorithm call interface for the reinforcement learning module, and obtains relevant information of the external simulation environment by calling the interface of the environment interaction layer. The environment interaction layer obtains environmental data from the external task simulation module and returns it to the algorithm interaction layer. Among them, the target custom function is also considered in the semantic interaction layer, which is also to realize that different users have special requirements for the reward value or reward function in the scene, so the semantic interaction layer can support the customization of the function. Compared with the related technology that can only develop the simulation application system corresponding to each task through different customization, the present application can adapt to the simulation environment required by different tasks, avoid the large resource consumption in the development of simulation application software, thereby improving the application efficiency of reinforcement learning in different task fields.

[0118] In some embodiments, see Figure 4 , Figure 4is a flow chart of the intelligent simulation interaction method based on reinforcement learning provided in the embodiment of the present application. The method is applied to the target decision platform in the intelligent simulation interaction system based on reinforcement learning provided in the above embodiment. The method may specifically include but is not limited to steps S410 to S460. Figure 4 These six steps are introduced in detail.

[0119] Step S410, receiving algorithm configuration information and environment configuration information of the target task sent by the interactive client;

[0120] Step S420, receiving candidate environment situation data sent by the task simulation module according to the task agent attribute information and the initial agent action information;

[0121] Step S430, comparing the agent state according to the candidate environment situation data and the task target information to obtain a state comparison result;

[0122] Step S440, if the state comparison result indicates that the state information of the multiple target intelligent agent objects corresponding to the candidate environment situation data does not conform to the intelligent agent target state information, action decisions are made for the multiple target intelligent agent objects according to the task target information and the candidate environment situation data to obtain intelligent agent decision data, and the intelligent agent decision data is sent to the task simulation module;

[0123] Step S450, receiving the adjusted environment situation data obtained by the task simulation module after adjusting the agent actions of multiple target agent objects in the simulation environment according to the agent decision data;

[0124] Step S460, performing agent state comparison again based on the adjusted environmental situation data and the task target information to update the state comparison result, until the state comparison result indicates that the state information of multiple target agent objects corresponding to the adjusted environmental situation data is consistent with the agent target state information.

[0125] The environment configuration information includes the task agent attribute information, the initial agent action information and the task target information, and the task target information is used to indicate the agent target state information when the target task is completed. The candidate environment situation data is used to characterize the information of each target agent related to the target task after real-time action in the simulation environment.

[0126] The intelligent simulation interaction method based on reinforcement learning provided in the embodiment of the present application provides a solution for the interaction between reinforcement learning training and an external simulation environment. The environmental interaction module designed by the present invention realizes the decoupling between the reinforcement learning algorithm and the external simulation environment through the division of the semantic interaction layer, the environmental interaction layer and the algorithm interaction layer, and provides a pluggable simulation environment access solution. It can also realize the interaction between different simulation services and the reinforcement learning algorithm, overcome the problem that the model training cannot use the external simulation service, and is conducive to the integrated development and management of the reinforcement learning algorithm. Among them, through the configuration of the semantic interaction layer, the business needs of multiple fields can be realized, such as radar intelligent detection, drone confrontation and other exercise scenarios. In addition, the environmental interaction module and the reinforcement learning algorithm module together constitute a target decision-making platform, which can adapt to the simulation environment required by different tasks, avoid the large resource consumption in the development of simulation application software, and thus improve the application efficiency of reinforcement learning in different task fields.

[0127] It should be noted that the intelligent simulation interaction system based on reinforcement learning in the embodiment of the present application is used to implement the intelligent simulation interaction method based on reinforcement learning in the above-mentioned embodiment. The intelligent simulation interaction method based on reinforcement learning in the embodiment of the present application corresponds to the aforementioned intelligent simulation interaction system based on reinforcement learning. For the specific processing process, please refer to the aforementioned intelligent simulation interaction system based on reinforcement learning, which will not be repeated here.

[0128] The embodiment of the present application also provides a computer device, which includes: at least one memory, at least one processor, and at least one computer program, wherein the at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement the intelligent simulation interaction method based on reinforcement learning in the above embodiment. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0129] See also Figure 5 , Figure 5 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes:

[0130] The processor 510 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0131] The memory 520 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 520 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 520, and the processor 510 calls and executes the intelligent simulation interaction method based on reinforcement learning in the embodiment of this application;

[0132] Input / output interface 530, used to implement information input and output;

[0133] Communication interface 540, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0134] bus 550 , which transmits information between the various components of the device (e.g., processor 510 , memory 520 , input / output interface 530 , and communication interface 540 );

[0135] The processor 510 , the memory 520 , the input / output interface 530 and the communication interface 540 are connected to each other in communication within the device via the bus 550 .

[0136] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program is used to enable a computer to execute the intelligent simulation interaction method based on reinforcement learning in the above embodiment.

[0137] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0138] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0139] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0140] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0142] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0143] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0144] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0148] The preferred embodiments of the present invention are described with reference to the accompanying drawings, but the scope of the present invention is not limited by the above. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. An intelligent simulation interactive system based on reinforcement learning, characterized in that: The system includes an interactive client, a task simulation module and a target decision platform, wherein the target decision platform includes a reinforcement learning algorithm module and an environment interaction module, wherein the reinforcement learning algorithm module is communicatively connected to the interactive client, and the environment interaction module is communicatively connected to the interactive client and the task simulation module respectively; The interactive client is used to send the algorithm configuration information of the target task to the reinforcement learning algorithm module, and send the environment configuration information of the target task to the environment interaction module; wherein the environment configuration information includes task agent attribute information, initial agent action information and task target information, and the task target information is used to indicate the agent target state information when the target task is completed; The task simulation module is used to perform physical mapping in the simulation environment according to the received task agent attribute information and initial agent action information, and determine multiple target agent objects of the target task in the simulation environment; the task simulation module is also used to obtain information after each target agent object acts in the simulation environment in real time, obtain candidate environment situation data, and send the candidate environment situation data to the environment interaction module; The environment interaction module is used to compare the agent state according to the candidate environment situation data and the task target information to obtain the state comparison result; if the state comparison result indicates that the state information of the multiple target agent objects corresponding to the candidate environment situation data does not conform to the agent target state information, the candidate environment situation data is sent to the reinforcement learning algorithm module; wherein, the environment interaction module is used to compare the agent state according to the candidate environment situation data and the task target information to obtain the state comparison result, specifically including: the environment interaction module is used to receive the candidate environment situation data from the task simulation module, and convert the data format of the candidate environment situation data to obtain the converted environment situation data; the environment interaction module is also used to compare the agent state according to the converted environment situation data and the task target information to obtain the state comparison result; The reinforcement learning algorithm module is used to obtain the environment configuration information and the candidate environment situation data from the environment interaction module; the reinforcement learning algorithm module is also used to make action decisions for the multiple target intelligent agents according to the task target information and the candidate environment situation data, obtain intelligent agent decision data, and send the intelligent agent decision data to the environment interaction module; The task simulation module is also used to adjust the agent actions of multiple target agent entities in the simulation environment according to the agent decision data to obtain adjusted environmental situation data, and send the adjusted environmental situation data to the environment interaction module to perform agent state comparison again to update the state comparison result until the state comparison result indicates that the state information of the multiple target agent entities corresponding to the adjusted environmental situation data is consistent with the agent target state information.

2. The system according to claim 1, characterized in that The target decision platform includes a semantic interaction layer, which is respectively connected to the interactive client, the reinforcement learning algorithm module and the environment interaction module in communication; The semantic interaction layer is used to receive the algorithm configuration information and the environment configuration information from the interactive client, wherein the algorithm configuration information includes the scene configuration information of the reinforcement learning scene, and the scene configuration information is used to guide the reinforcement learning algorithm module to make action decisions for the plurality of target intelligent entities according to the task target information and the environmental situation data received in real time; The semantic interaction layer is also used to receive a target custom function related to the task target information from the interactive client when the candidate environment situation data does not contain task reward sub-data, and send the target custom function to the reinforcement learning algorithm module, the target custom function is used to calculate the task reward according to the candidate environment situation data to obtain the task reward sub-data, and the task reward sub-data is used to generate the agent decision data.

3. The system according to claim 1, characterized in that The target decision platform includes an environment interaction layer, the environment interaction layer is communicatively connected with the environment interaction module and the task simulation module, and the environment interaction layer includes a first connection interface, an initialization interface and an interaction sending interface; The first connection interface is used to connect to the task simulation module; The initialization interface is used to initialize the task simulation module when the task simulation module completes configuration input; the initialization interface is also used to restore the task simulation module to the initialization state at startup after determining that the state information of the plurality of target intelligent agent objects corresponding to the adjusted environmental situation data conforms to the intelligent agent target state information; The interactive sending interface is used to call the interactive class interface of the task simulation module to send the agent decision data obtained by the environment interaction module from the reinforcement learning algorithm module to the task simulation module; the interactive sending interface is also used to send the adjusted environmental situation data obtained after adjustment of the task simulation module to the environment interaction module.

4. The system according to claim 3, characterized in that The environment interaction layer also includes a log management interface and a cross-language calling interface; The log management interface is used to record all data interacting between the task simulation module and the environment interaction module; The cross-language calling interface is used to implement interface calling and data exchange between different programming languages ​​connected to the target decision platform.

5. The system according to claim 3, characterized in that The target decision platform also includes an algorithm interaction layer, which is communicatively connected with the reinforcement learning algorithm module and the environment interaction module, and the algorithm interaction layer includes a second connection interface, a reset interface, and an acquisition interface; The second connection interface is used to establish a communication connection between the algorithm interaction layer and the environment interaction layer before the reinforcement learning training is started; the second connection interface is also used to call the first connection interface; The reset interface is used to call the first connection interface and reset the task simulation module after determining that the state information of the plurality of target intelligent agent objects corresponding to the adjusted environmental situation data conforms to the intelligent agent target state information, so as to perform the next round of reinforcement learning training; The acquisition interface is used to obtain the agent decision data output by the reinforcement learning algorithm module, and the acquisition interface is also used to obtain the candidate environment situation data output by the environment simulation module from the environment interaction module.

6. The system according to claim 1, characterized in that After updating the state comparison result until the state comparison result indicates that the state information of the plurality of target intelligent agent objects corresponding to the adjusted environmental situation data conforms to the intelligent agent target state information, the environmental interaction module is further used to extract reward data from all environmental situation data received by the environmental interaction module to obtain a reward data sequence; The environment interaction module is also used to generate a reward process curve according to the reward data sequence, and send the reward process curve to the interactive client.

7. An intelligent simulation interaction method based on reinforcement learning, characterized in that: A target decision platform applied to the intelligent simulation interactive system based on reinforcement learning according to any one of claims 1 to 6, the method comprising: Receive algorithm configuration information and environment configuration information of the target task sent by the interactive client; wherein the environment configuration information includes task agent attribute information, initial agent action information and task target information, and the task target information is used to indicate the agent target state information when completing the target task; Receiving candidate environment situation data sent by the task simulation module according to the task agent attribute information and the initial agent action information, wherein the candidate environment situation data is used to represent information after each target agent object related to the target task acts in real time in the simulation environment; Comparing the state of the intelligent body with the candidate environmental situation data and the task target information to obtain a state comparison result; the comparing the state of the intelligent body with the candidate environmental situation data and the task target information to obtain a state comparison result includes: converting the data format of the candidate environmental situation data to obtain converted environmental situation data; comparing the state of the intelligent body with the converted environmental situation data and the task target information to obtain the state comparison result; If the state comparison result indicates that the state information of the plurality of target intelligent agent objects corresponding to the candidate environmental situation data does not conform to the intelligent agent target state information, action decisions are made for the plurality of target intelligent agent objects according to the task target information and the candidate environmental situation data to obtain intelligent agent decision data, and the intelligent agent decision data is sent to the task simulation module; Receiving the adjusted environment situation data obtained by the task simulation module after adjusting the agent actions of the plurality of target agent entities in the simulation environment according to the agent decision data; The agent state comparison is performed again based on the adjusted environmental situation data and the task target information to update the state comparison result, until the state comparison result indicates that the state information of the multiple target agent objects corresponding to the adjusted environmental situation data is consistent with the agent target state information.

8. A computer device, characterized in that: include: at least one memory; at least one processor; at least one computer program; The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement: The method as claimed in claim 7.

9. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, and the computer program is used to cause a computer to execute: The method as claimed in claim 7.

Images