A method and device for designing a visualized drag reinforcement learning agent

Abstract

The application discloses a visual drag reinforcement learning agent design method and device, and relates to the technical field of artificial intelligence. The method comprises the following steps: in response to a component retrieval configuration operation, determining a to-be-used component from a pre-built component library and setting component configuration parameters; in response to a drag and splice operation, loading the to-be-used component as a component node of an agent design canvas to generate an agent component topology graph; obtaining a to-be-adapted scenario scheme and performing scene mapping, environment interaction mapping and rule binding to obtain a scenario adaptation relationship; based on the agent component topology graph, the component configuration parameters and the scenario adaptation relationship, generating a design-state agent and generating interface adaptation information; and in response to a training start operation, connecting the interface adaptation information to an agent training engine and a simulation environment to start training. The application reduces the technical threshold, improves the design and scenario adaptation rate, realizes the connection between the design and training links, and thus improves the overall development iteration efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for designing a visual drag-and-drop reinforcement learning agent. Background Technology

[0002] As reinforcement learning technology is increasingly applied in scenarios such as simulation, intelligent decision-making, unmanned system control, resource scheduling, and complex adversarial scenarios, the design, training, and iteration of reinforcement learning agents are gradually becoming crucial aspects of the practical application of artificial intelligence. Reinforcement learning agents typically need to continuously interact with the environment in a simulation environment, generate action commands based on the environment's state, and update their strategies based on reward information from environmental feedback, thereby gradually optimizing their behavioral decision-making capabilities.

[0003] In current technologies, the design process of reinforcement learning agents typically relies on professional algorithm engineers to manually code the code. Specifically, these engineers manually build core functional modules from the ground up, including situational inputs, neural networks, action outputs, and reward functions. They then use code to establish logical connections, data transfer, and parameter configurations between these modules, resulting in the agent model. When the agent needs to adapt to different simulation scenarios, engineers must extract scenario data, reward rules, decision-making methods, and statistical indicators from the proposed scenarios and adapt them to the agent model using additional code.

[0004] It can be seen that the current design of reinforcement learning agents relies entirely on manual coding, which has a high technical threshold, low design speed and low scenario adaptation speed, and the design and training stages are separated, resulting in low overall development iteration efficiency. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a visual drag-and-drop reinforcement learning agent design method and apparatus. By integrating visual and modular agent design and training, this application lowers the technical threshold, increases the design speed and scenario adaptation speed, and achieves seamless integration of the design and training phases, thereby significantly improving the overall development and iteration efficiency of reinforcement learning agents.

[0006] The embodiments of this application disclose the following technical solutions: In a first aspect, embodiments of this application provide a method for designing a reinforcement learning agent using a visual drag-and-drop interface, comprising: In response to the component retrieval and configuration operation, a component to be used that matches the reinforcement learning agent design task is determined from the pre-built component library, and the component configuration parameters of the component to be used are set; wherein, the pre-built component library includes standardized components obtained by encapsulating the functions of the reinforcement learning agent; A smart agent design canvas is generated in the visual operation interface. In response to the drag and splice operation of the component to be used, the component to be used is loaded as a component node in the smart agent design canvas, and the logical relationship and data flow relationship between the component nodes are established to generate a smart agent component topology diagram. Obtain the adaptation scenario scheme corresponding to the topology diagram of the intelligent agent component, and perform scene mapping, environment interaction mapping and rule binding on the component nodes in the topology diagram of the intelligent agent component according to the adaptation scenario scheme to obtain the scenario adaptation relationship; Based on the topology of the agent components, the configuration parameters of the components, and the hypothetical adaptation relationship, a design-state agent is generated, and interface adaptation information for the design-state agent to interface with the agent training engine and simulation environment is generated. In response to the training start operation for the designed agent, the designed agent is connected to the agent training engine and the simulation environment through the interface adaptation information to start the reinforcement learning training of the designed agent.

[0007] In one possible implementation, the standardized components in the pre-built component library include: simulation components, situation input components, neural network components, action output components, reward function components, data statistics components, and evaluation metric components; wherein, the simulation component is used to configure the simulation environment mirror path and associated projects; the situation input component includes a global statistical situation sub-component, an entity situation sub-component, and a spatial situation sub-component; the neural network component includes an encoder sub-component, an aggregator sub-component, a decoder sub-component, and a value function sub-component; the action output component includes a predicate sub-component, a subject sub-component, an object sub-component, and a parameter sub-component; the reward function component is used to define the reward evaluation rules for agent behavior; the data statistics component is used to provide data statistics and analysis capabilities; and the evaluation metric component is used to quantitatively evaluate the training effect and performance of the agent; and each of the standardized components is configured with unique identification information and standardized input / output interfaces.

[0008] In one possible implementation, the step of responding to the component retrieval configuration operation by determining a component to be used that matches the reinforcement learning agent design task from a pre-built component library and setting the component configuration parameters of the component to be used includes: In response to the component retrieval operation, generate component retrieval conditions based on at least one of the component type, functional keywords, input / output interface requirements, and algorithm requirements corresponding to the reinforcement learning agent design task. Based on the component retrieval criteria, candidate components that match the reinforcement learning agent design task are filtered from the pre-built component library, and the component type, input / output interface and configurable parameters of the candidate components are displayed. In response to the component selection operation, the selected candidate component is instantiated as the component to be used; In response to the parameter configuration operation, the component configuration parameters of the component to be used are set; wherein, the component configuration parameters include one or more of the following: index name, activation status, feature information, network structure parameters, action instruction parameters, and reward quantification parameters.

[0009] In one possible implementation, the method further includes: If no candidate component matching the reinforcement learning agent design task is found in the pre-built component library based on the component retrieval conditions, the component template corresponding to the category of the custom component is invoked in response to the component creation operation. Configure the basic information, input / output interfaces, parameter items, and activation status of the custom component based on the component template; The custom component is validated for format and interface. Once the format and interface validations are passed, the custom component is added to the pre-built component library. Based on the component retrieval criteria, candidate components matching the reinforcement learning agent design task are screened again from the pre-built component library.

[0010] In one possible implementation, the step of loading the component to be used as a component node in the agent design canvas in response to drag-and-drop and splicing operations on the component to be used, and establishing logical relationships and data flow relationships between the component nodes to generate an agent component topology graph, includes: In response to a drag-and-drop operation on the component to be used, a component node corresponding to the component to be used is generated in the agent design canvas; wherein, the component node records a node identifier, a component identifier, a node position, an input interface, and an output interface; In response to the splicing operation on the component nodes, the output interface of the first component node as the starting point of the connection and the input interface of the second component node as the ending point of the connection are determined. Based on the output interface of the first component node and the input interface of the second component node, the legality of the connection between the first component node and the second component node is verified. When the connection validity verification passes, a directed connection is established from the first component node to the second component node, and corresponding edge information is generated; wherein, the edge information includes the starting node identifier, the ending node identifier, the starting interface, the ending interface, the data flow direction, and the logical dependency type; A topology graph of the intelligent agent components is generated based on the component nodes, the directed connections, and the edge information corresponding to the directed connections. The method further includes: When the docking validity check fails, a docking error message is output and the establishment of a directed connection from the first component node to the second component node is prevented.

[0011] In one possible implementation, after generating the agent component topology graph based on the component nodes, the directed connections, and the edge information corresponding to the directed connections, the method further includes: The topology validity of the agent component topology graph is verified. This verification includes: determining whether the agent component topology graph contains situation input component nodes, neural network component nodes, and action output component nodes used to form the operational logic of the reinforcement learning agent; determining whether a data flow path from situation data input, feature extraction and fusion to action command output is formed between the situation input component nodes, the neural network component nodes, and the action output component nodes; determining whether there are unconnected isolated component nodes, invalid directed connections, or connections that do not conform to logical dependency types in the agent component topology graph; and determining whether the required component configuration parameters of the component nodes are configured completely. The step of performing scene mapping, environment interaction mapping, and rule binding on the component nodes in the intelligent agent component topology graph according to the scenario to be adapted, to obtain the scenario adaptation relationship, includes: If the topology legality verification passes, according to the proposed adaptation scenario, the component nodes in the intelligent agent component topology graph are mapped to the scene, mapped to the environment interaction, and bound to the rules to obtain the proposed adaptation relationship. The method further includes: If the topology validity check fails, locate the abnormal component node or abnormal directed connection in the agent design canvas and output a topology error message.

[0012] In one possible implementation, the scenario to be adapted includes basic scenario information, predefined scenario data, operating mechanism, reward function adaptation rules, and data statistics rules; the step of obtaining the scenario to be adapted corresponding to the topology graph of the agent component, and performing scenario mapping, environment interaction mapping, and rule binding on the component nodes in the topology graph of the agent component according to the scenario to be adapted, to obtain the scenario adaptation relationship, includes: In response to the scenario selection operation, obtain the scenario to be adapted associated with the topology graph of the intelligent agent component; Map the situation data source, environmental parameters and scene triggering conditions in the predefined scenario data to the corresponding situation input component node or simulation component node in the intelligent agent component topology diagram; Based on the decision time, running time and auxiliary rules in the operating mechanism, the interaction cycle and interaction rules between the design-state intelligent agent and the simulation environment corresponding to the intelligent agent component topology diagram are mapped to obtain the interaction cycle and interaction rules between the intelligent agent component topology diagram and the simulation environment. The reward function adaptation rule is bound to the reward function component node in the agent component topology graph, and the data statistics rule is bound to the data statistics component node or evaluation index component node in the agent component topology graph to obtain the desired adaptation relationship.

[0013] In one possible implementation, the step of generating a design-state intelligent agent based on the intelligent agent component topology graph, the component configuration parameters, and the hypothetical adaptation relationship, and generating interface adaptation information for the design-state intelligent agent to interface with the intelligent agent training engine and simulation environment, includes: Based on the logical relationships and data flow relationships in the intelligent agent component topology diagram, extract the component identifier, component configuration parameters, connection relationship information, and hypothetical adaptation relationship of the component node; Based on the extracted component identifier, component configuration parameters, connection relationship information, and hypothetical adaptation relationship, standardized agent description information is generated, and a design-state agent is generated according to the standardized agent description information; wherein, the standardized agent description information includes at least one of: a standardized agent description file and executable construction instructions; Generate interface adaptation information for the designed intelligent agent to interface with the intelligent agent training engine and simulation environment; wherein, the interface adaptation information includes at least one of the following: state input interface information, action output interface information, reward feedback interface information, training control interface information, and evaluation data acquisition interface information.

[0014] In one possible implementation, after connecting the designed agent to the agent training engine and the simulation environment via the interface adaptation information to initiate reinforcement learning training of the designed agent, the method further includes: During the reinforcement learning training process of the designed intelligent agent, training-related data is collected through data statistics components and / or evaluation index components associated with the designed intelligent agent; wherein, the training-related data includes: training data, reward value, resource consumption data and evaluation index data; The collected training-related data are displayed using visual curves or statistical panels.

[0015] Secondly, embodiments of this application provide a visual drag-and-drop reinforcement learning agent design device, comprising: The retrieval configuration module is used to respond to the component retrieval configuration operation, determine the component to be used that matches the reinforcement learning agent design task from the pre-built component library, and set the component configuration parameters of the component to be used; wherein, the pre-built component library includes standardized components obtained by encapsulating the functions of the reinforcement learning agent; The drag-and-drop splicing module is used to generate an intelligent agent design canvas in the visual operation interface. In response to the drag and splicing operation of the component to be used, the component to be used is loaded as a component node in the intelligent agent design canvas, and the logical relationship and data flow relationship between the component nodes are established to generate an intelligent agent component topology diagram. The scenario adaptation module is used to obtain the scenario to be adapted corresponding to the topology diagram of the intelligent agent component, and to perform scene mapping, environment interaction mapping and rule binding on the component nodes in the topology diagram of the intelligent agent component according to the scenario to be adapted, so as to obtain the scenario adaptation relationship. The design generation module is used to generate a design-state intelligent agent based on the intelligent agent component topology diagram, the component configuration parameters and the hypothetical adaptation relationship, and to generate interface adaptation information for the design-state intelligent agent to interface with the intelligent agent training engine and simulation environment. The reinforcement training module is used to respond to the training start operation for the designed agent by connecting the designed agent to the agent training engine and the simulation environment through the interface adaptation information, so as to start the reinforcement learning training of the designed agent.

[0016] Beneficial effects: The core functions of reinforcement learning agents are encapsulated into standardized components and uniformly managed in a pre-built component library, supporting retrieval, configuration, and reuse. This eliminates the need for low-level coding, lowering the technical threshold and increasing design speed. Component structures and data flow paths are intuitively constructed through a visual drag-and-drop canvas, and design defects are promptly identified through legality checks, reducing debugging costs. Scene mapping, environmental interaction mapping, and rule binding are automatically completed through scenario planning, achieving standardized scene adaptation. Design-state agents and interface adaptation information are automatically generated based on topology diagrams and configuration parameters, achieving seamless integration between design and training. During training, visual data collection and display support rapid iterative adjustments, comprehensively improving the efficiency of agent development iteration. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a visual drag-and-drop reinforcement learning agent design method provided in this application embodiment; Figure 2 A schematic diagram illustrating the process of retrieving and obtaining the component to be used, provided for an embodiment of this application; Figure 3 An example diagram of a visual operation interface related to the retrieval of components to be used, provided for embodiments of this application; Figure 4 An example diagram of a visual operation interface including an intelligent agent design canvas provided for embodiments of this application; Figure 5 A schematic diagram of the process for generating a topology diagram of intelligent agent components provided in an embodiment of this application; Figure 6 This is a schematic diagram of the process for obtaining the designed intelligent agent provided in the embodiments of this application; Figure 7 This is a schematic diagram of a visual drag-and-drop reinforcement learning agent design device provided in an embodiment of this application. Detailed Implementation

[0019] As described earlier, current reinforcement learning agent design typically relies on professional algorithm engineers manually writing code. Engineers need to repeatedly develop functional modules such as situation input, neural networks, action output, reward functions, and training statistics for different tasks, and combine and configure the parameters of each module through code. When it is necessary to adapt to different scenarios, it is also necessary to manually extract scene data, operating mechanisms, and reward rules, and write adaptation code to adapt the scenario to the agent model.

[0020] Furthermore, after the agent design is completed, existing technologies still require manual coding of interface code to connect the agent to an external agent training engine and simulation environment to initiate the reinforcement learning training process. During training, if problems are found in the agent structure, network parameters, action output rules, or reward function configuration, it is usually necessary to go back to the code level to modify the relevant modules, reconnect to the training environment, and start training again. This results in a high degree of manual involvement in the design, debugging, and training iteration processes.

[0021] Therefore, it can be seen that in the current reinforcement learning agent design methods, the design of agents relies entirely on manual coding, which has a high technical threshold and lacks visual design methods, resulting in a low agent design speed. The functional modules of agents lack standardized and component-based management mechanisms, making it difficult to reuse similar functional modules, which easily leads to redundant development and a large amount of repetitive R&D work, resulting in low development efficiency. The scenario plan and agent design process are disconnected, and scenario data, operating mechanisms, reward rules, and statistical indicators need to be manually adapted, resulting in low scenario adaptation efficiency. In addition, the agent design stage is separated from the training stage. After the design is completed, manual docking with the training engine and simulation environment is still required. For example, the effect monitoring data during the training process needs to be extracted separately from the training engine and simulation environment. If design problems are found, it is necessary to go back to the code level to modify the agent model and then manually dock with the training environment again to start training, which makes it difficult to achieve rapid training, debugging, and iterative optimization.

[0022] This application provides a visual drag-and-drop reinforcement learning agent design method, including: in response to a component retrieval and configuration operation, determining a component to be used that matches the reinforcement learning agent design task from a pre-built component library, and setting the component configuration parameters of the component to be used; generating an agent design canvas in a visual operation interface; in response to drag-and-drop and splicing operations on the component to be used, loading the component to be used as a component node in the agent design canvas, establishing logical relationships and data flow relationships between component nodes, and generating an agent component topology graph; obtaining the topology graph of the agent component. The application identifies the corresponding scenario to be adapted and, based on this scenario, performs scene mapping, environment interaction mapping, and rule binding on the component nodes in the agent component topology diagram to obtain the scenario adaptation relationship. Based on the agent component topology diagram, component configuration parameters, and scenario adaptation relationship, a design-state agent is generated, and interface adaptation information for connecting to the agent training engine and simulation environment is generated for the design-state agent. In response to the training start operation for the design-state agent, the design-state agent is connected to the agent training engine and simulation environment through the interface adaptation information to initiate reinforcement learning training. This application, through a visual, component-based integrated approach to agent design and training, lowers the technical threshold, improves the design speed and scenario adaptation speed, and achieves seamless integration of the design and training stages, thereby significantly improving the overall development and iteration efficiency of reinforcement learning agents.

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0024] The collection and processing of relevant data (including but not limited to experimental data, test data, simulation data, user data, etc.) involved in this application shall strictly comply with the requirements of national laws and regulations when applied in the following embodiments, obtain the informed consent or separate consent of the subject obtaining the data information, and carry out data use and processing within the scope of laws and regulations and the authorization of the subject.

[0025] Method Implementation Examples: The following is combined Figures 1-6 This application provides a detailed description of a visual drag-and-drop reinforcement learning agent design method based on its embodiments.

[0026] like Figure 1 As shown in the embodiments of this application, a visual drag-and-drop reinforcement learning agent design method may include the following steps: S101. In response to the component retrieval and configuration operation, determine the component to be used that matches the reinforcement learning agent design task from the pre-built component library, and set the component configuration parameters of the component to be used.

[0027] The pre-built component library includes standardized components derived from the functional encapsulation of reinforcement learning agents. Standardized components refer to reusable functional components formed by encapsulating functional units in reinforcement learning agents according to a unified encapsulation specification.

[0028] Each standardized component is equipped with unique identification information and standardized input / output interfaces.

[0029] For example, each standardized component can be configured with unique identification information, component category, component description, input interface, output interface, configurable parameters, and enabled status.

[0030] In this embodiment of the application, the pre-built component library may include simulation components, situation input components, neural network components, action output components, reward function components, data statistics components, and evaluation index components.

[0031] The simulation component is used to configure the simulation environment, which can include configuration parameters such as the simulation environment image path and associated projects. Specifically, the simulation component is used to configure the simulation environment image path and associated projects.

[0032] The situation input component is used to standardize the input of situation data. Specifically, the situation input component includes a global statistical situation sub-component, an entity situation sub-component, and a spatial situation sub-component. The global statistical situation sub-component is used to obtain global situation information, the entity situation sub-component is used to obtain situation information for a specified unit type or entity object, and the spatial situation sub-component is used to obtain spatial environment information. Configurable parameters for the situation input component can include index name, situation name, description, whether it is enabled, and feature information.

[0033] The neural network component is used to extract, fuse, decode, and evaluate the value of the data output by the situation input component. Specifically, the neural network component includes an encoder sub-component, an aggregator sub-component, a decoder sub-component, and a value function sub-component. The encoder sub-component can include a global statistical situation encoder, an entity situation encoder, and a spatial situation encoder; the aggregator sub-component can include a GRU aggregator, a Dense aggregator, and an LSTM aggregator; and the decoder sub-component can include a classification decoder, a continuous decoder, and an unordered multi-select decoder. Configurable parameters for the neural network component can include the number of network layers, the number of neurons, the type of activation function, and the feature fusion method.

[0034] The action output component converts the computation results of the neural network components into standardized agent action instructions. Specifically, the action output component includes a predicate sub-component, a subject sub-component, an object sub-component, and a parameter sub-component. The subject sub-component defines the agent / entity object that performs the action; the predicate sub-component characterizes the type of action performed by the agent, which can include various decision-making action instructions such as maneuvering, attacking, reconnaissance, and evading; the object sub-component specifies the target object of the action, which can be configured as enemy units, positions, airspace, resources, etc.; the parameter sub-component supplements the constraints and quantification conditions for action execution, and can include numerical range configurations such as action distance, execution duration, attack intensity, and movement speed. Configurable parameters of the action output component can include action instruction format, output dimension, and parameter value range.

[0035] The reward function component defines the reward evaluation rules for agent behavior, providing a basis for reward and punishment in the optimization of reinforcement learning strategies. For example, the reward function component can include rules for enemy damage rewards, friendly damage penalties, enemy kill rewards, and friendly kill penalties.

[0036] The data statistics component provides data statistics and analysis capabilities, offering data support for training performance analysis and strategy optimization. The data statistics component allows configuration of data collection items, statistical periods, and result output formats during the training process.

[0037] The evaluation metrics component is used to quantitatively evaluate the training effectiveness and performance of the agent. For example, the evaluation metrics component may include metrics such as enemy casualties, friendly casualties, ammunition consumption, mission completion rate, and average reward value.

[0038] To make it easier to understand, the following will be combined with... Figure 2 and Figure 3 This section details how to obtain the components to be used.

[0039] S201. In response to the component retrieval operation, generate component retrieval conditions based on at least one of the component type, functional keywords, input / output interface requirements, and algorithm requirements corresponding to the reinforcement learning agent design task.

[0040] The component retrieval operation is a user-triggered operation used to retrieve components for use. For example, a user can open the component management interface in the visual operation interface and enter retrieval elements in the corresponding prompt box, such as at least one of the following: component type, functional keywords, input / output interface requirements, and algorithm requirements corresponding to the reinforcement learning agent design task.

[0041] The component type can be used to limit the category to which the component to be retrieved belongs. For example, when a user needs to configure the situational awareness capability of an agent, they can select the situational input component; when a user needs to build the network structure of an agent, they can select the neural network component; when a user needs to configure the decision output method of an agent, they can select the action output component.

[0042] Functional keywords can be used to define the specific functions of a component. For example, users can enter keywords such as "spatial situation", "entity situation", "GRU aggregation", "classification decoding", "continuous decoding", "enemy damage reward", and "ammunition consumption statistics" to search for components with the corresponding functions.

[0043] Input / output interface requirements can be used to define the interface specifications for the input and output data required by a component. For example, if a reinforcement learning agent design task requires a component to receive a spatial situation matrix and output a feature vector of fixed dimensions, then this input / output interface requirement can be used as part of the search criteria to filter components that match the interface specifications.

[0044] Algorithm requirements can be used to define the algorithmic capabilities needed to construct a neural network architecture or reinforcement learning agent. For example, if the reinforcement learning agent design task requires the use of temporal feature processing capabilities, the algorithmic requirements may include GRU, LSTM, etc.; if discrete actions need to be output, the algorithmic requirements may include a classification decoder; if continuous control quantities need to be output, the algorithmic requirements may include a continuous decoder.

[0045] Among them, the component retrieval conditions refer to the multi-dimensional filtering rules generated in response to the component retrieval operation. These rules may include at least one of the following: component type, functional keywords, input / output interface requirements, and algorithm requirements. They are used to accurately locate functional components that meet the requirements of the reinforcement learning agent design task.

[0046] Specifically, in response to the component retrieval operation, the system automatically receives and parses the retrieval elements input by the user (i.e., at least one of the component type, functional keywords, input / output interface requirements, and algorithm requirements corresponding to the reinforcement learning agent design task) to generate component retrieval conditions.

[0047] S202. Based on the component retrieval criteria, filter candidate components that match the reinforcement learning agent design task from the pre-built component library, and display the component type, input / output interface and configurable parameters of the candidate components.

[0048] Specifically, the system filters standardized components from a pre-built component library based on component search criteria. The filtering process may include at least one of component category matching, keyword matching, interface matching, and algorithm capability matching.

[0049] For example, when the component search criteria include "neural network component" and "GRU aggregation", the system can filter aggregator components under the neural network component category, and further filter components whose functional description or algorithm type contains GRU, obtaining GRU aggregator components as candidate components. As another example, when the component search criteria include "spatial situation" and "output feature vector", the system can filter components from the situation input components and neural network components that can process spatial situation data and output feature vectors, such as spatial situation sub-components or spatial situation encoder components.

[0050] Furthermore, candidate components that match the reinforcement learning agent design task are selected. The component type, input / output interfaces, and configurable parameters of the candidate components can be displayed in the preset candidate component display area in the visual operation interface. The component name, component identifier, and functional description of the candidate components can also be displayed.

[0051] The input and output interfaces may include: input interface name, output interface model, input interface type, output interface type, data type, data dimension, field identifier, and data format.

[0052] In this embodiment of the application, by displaying the component type, input / output interface and configurable parameters of the candidate components, users can intuitively determine whether the candidate components meet the requirements of the reinforcement learning agent design task, thereby reducing the difficulty of component selection.

[0053] In one possible implementation, if no candidate component matching the reinforcement learning agent design task is found in the pre-built component library based on the component retrieval criteria, in response to the component creation operation, a component template corresponding to the category of the custom component is invoked; the basic information, input / output interfaces, parameter items, and activation status of the custom component are configured based on the component template; the format and interface of the custom component are validated; if the format and interface validations pass, the custom component is added to the pre-built component library; and candidate components matching the reinforcement learning agent design task are screened again in the pre-built component library based on the component retrieval criteria.

[0054] Format validation can include component identifier uniqueness validation, parameter field integrity validation, parameter type validation, and parameter value range validation. Interface validation can include input interface field integrity validation, output interface field integrity validation, interface direction validation, and interface data format validation.

[0055] S203. In response to the component selection operation, the selected candidate component is instantiated as the component to be used.

[0056] The component selection operation is the user's action of choosing a target candidate component from the candidate component display area. For example, users can trigger the component selection operation by clicking the selection button in the candidate component list, double-clicking a candidate component item, or dragging the candidate component icon.

[0057] It should be noted that candidate components can be understood as standardized component templates in a pre-built component library, while components to be used can be understood as actual component instances used in the current reinforcement learning agent design task.

[0058] Specifically, the system can generate a component instance identifier for the component to be used, and copy or reference information such as the candidate component's component identifier, component type, input / output interface definition, default parameters, and functional description to that component instance. The component to be used is associated with the standardized components in the pre-built component library, but the parameter configuration of the component to be used can be modified independently of the component templates in the component library.

[0059] In this embodiment of the application, by instantiating the candidate component as the component to be used, the modification of the component parameters by the user in the current reinforcement learning agent design task can be avoided from affecting the standardized component template in the pre-built component library, thereby ensuring the reusability and stability of the components in the component library.

[0060] S204. In response to the parameter configuration operation, set the component configuration parameters of the component to be used.

[0061] Among these, parameter configuration operations are actions performed by the user to edit, confirm, or enable / disable parameters of the component to be used. For example, the user can fill in or modify the component configuration parameters of the component to be used in the component parameter configuration area.

[0062] The component configuration parameters include one or more of the following: index name, activation status, feature information, network structure parameters, action instruction parameters, and reward quantification parameters.

[0063] The index name identifies the data index or call name of the component to be used in the current reinforcement learning agent design task. The enabled status indicates whether the component to be used participates in subsequent agent component topology construction and design-state agent generation. Feature information defines the feature fields collected or output by the situation input component, such as global statistical features, entity state features, or spatial environment features. Network structure parameters configure the number of network layers, number of neurons, activation function type, feature fusion method, etc., of the neural network component. Action command parameters configure the command format, output dimension, and parameter value range of the action output component. Reward quantization parameters configure the reward and penalty rules, reward events, and reward weights corresponding to the reward function component.

[0064] For example, in response to the user's component selection operation (i.e., the user selects the GRU aggregator component), an instantiation operation is performed to determine that the component to be used is the "GRU aggregator component". The user can configure parameters such as the hidden layer dimension, number of network layers, activation function and whether to enable the GRU aggregator component in the component parameter configuration area of ​​the visual operation interface.

[0065] For example, in response to the user's component selection operation, an instantiation operation is performed to determine the component to be used as an "action output component". The user can configure its action predicate, action subject, action object, output dimension and parameter value range in the component parameter configuration area in the visual operation interface, so that the action output component can generate action instructions that can be recognized by the simulation environment according to a preset format.

[0066] Furthermore, after setting the component configuration parameters of the component to be used, the component configuration parameters of the component to be used can be saved, and the component to be used can be marked as loadable so that it can respond to drag and drop operations in subsequent steps and be loaded as a component node in the agent design canvas.

[0067] To make it easier to understand, the following will be combined with... Figure 3 Examples of visual operation interfaces related to the retrieval of components to be used are provided. Figure 3 As shown, the visual operation interface related to the retrieval of the component to be used is operation interface 300.

[0068] like Figure 3 As shown, the operation interface 300 includes a component classification area 310 on the left, a search condition input area 320 at the top, a candidate component display area 330 in the middle, and a component parameter configuration area 340 on the right or bottom.

[0069] The component category area 310 displays standardized components from a pre-built component library, categorized by component type. Examples include simulation components, situation input components, neural network components, action output components, reward function components, data statistics components, and evaluation metric components. Users can click on the corresponding component category to view the standardized components within that category.

[0070] The search criteria input area 320 is used to receive search elements such as component type, functional keywords, input / output interface requirements, and algorithm requirements input by the user. That is, when the user inputs search requirements in the search criteria input area 320, the component search operation is triggered. For example, the user can enter keywords such as "GRU," "spatial situation encoder," or "classification decoder," or select the component category or interface type from the drop-down list.

[0071] The candidate component display area 330 is used to display candidate components filtered based on component search criteria. Candidate components can be displayed in list format, with each row corresponding to one candidate component, showing information such as component name, component type, input interface, output interface, component description, and configurable parameters. For example, a user can select a component to use by clicking in the candidate component display area 330, thus triggering the component selection operation.

[0072] The component parameter configuration area 340 is used to display and edit the configuration parameters of the selected candidate component. For example, users can set the index name, activation status, feature information, number of network layers, number of neurons, activation function type, action instruction format, output dimension, parameter value range, and reward quantization parameters in this area, thus triggering the parameter configuration operation.

[0073] pass Figure 3 The visual interface shown allows users to retrieve, select, instantiate, and configure parameters of standardized components without manually writing code, providing components to be used in subsequent drag-and-drop assembly in the agent design canvas.

[0074] The above combination Figure 2 and Figure 3 This paper details the process of component retrieval and determination of the component to be used in the embodiments of this application. The following section continues with further examples. Figure 1 This application provides a detailed description of a visual drag-and-drop reinforcement learning agent design method.

[0075] S102. Generate an intelligent agent design canvas in the visual operation interface. In response to drag and stitch operations for the components to be used, load the components to be used as component nodes in the intelligent agent design canvas, establish logical relationships and data flow relationships between component nodes, and generate an intelligent agent component topology diagram.

[0076] The Agent Design Canvas is a graphical interface for visually designing reinforcement learning agents. Users can use drag-and-drop, move, delete, join, and connect components to lay out their layouts and establish relationships.

[0077] Among them, drag and drop and splicing operations refer to the interactive actions performed by users in the visual operation interface. Dragging refers to moving the component to be used from the component control toolbar to the specified position on the intelligent agent design canvas, and splicing refers to connecting the interfaces of two component nodes through a connection method to complete the logical connection between components.

[0078] Among them, component nodes refer to the visual instances of the components to be used generated in the agent design canvas. Each node fully retains the input and output interface definitions, configurable parameters and built-in functional attributes of the corresponding component, and is the basic unit that constitutes the agent component topology diagram.

[0079] Logical relationships refer to the business dependencies and function call relationships between component nodes. These relationships clarify the execution order of each functional unit and the upstream / downstream call logic during agent runtime, ensuring that the business processes of situation input, feature extraction, action output, and reward calculation conform to the operational logic of the reinforcement learning agent. Data flow relationships refer to the data transmission paths and data type conventions between component nodes. Based on the pre-configured standardized input / output interface definitions of each component, these relationships clarify how the output data of upstream components is transmitted to the input ports of downstream components, ensuring seamless data integration and eliminating format compatibility issues between different components.

[0080] Among them, the intelligent agent component topology diagram refers to the graphical intelligent agent architecture diagram composed of all component nodes and the connections between nodes on the intelligent agent design canvas, which fully records the layout position, association logic and data flow of all components.

[0081] To make it easier to understand, the following will be combined with... Figure 4 For example, we will introduce the intelligent agent design canvas.

[0082] like Figure 4 As shown, the visual operation interface 400 may include a component control toolbar 410 and an agent design canvas 420. The component control toolbar 410 is used to display the components to be used; the agent design canvas 420 is used to display component nodes, the connections between component nodes, and the overall structure of the agent.

[0083] To make it easier to understand, the following will be combined with... Figure 5 This document details how the topology diagram of the intelligent agent components is generated in the embodiments of this application.

[0084] S501, In response to a drag-and-drop operation on the component to be used, generate a component node corresponding to the component to be used in the agent design canvas.

[0085] Among them, the component node can record the node identifier, component identifier, node position, input interface and output interface.

[0086] Node identifiers are used to uniquely identify nodes in the agent design canvas, component identifiers are used to associate standardized components in the pre-built component library, node positions are used to record the display position of component nodes in the agent design canvas, and input and output interfaces are used to define the data input and data output relationships between component nodes.

[0087] S502, In response to the splicing operation for component nodes, determine the output interface of the first component node as the starting point of the connection, and the input interface of the second component node as the ending point of the connection.

[0088] S503. Based on the output interface of the first component node and the input interface of the second component node, perform a connection validity check on the first component node and the second component node.

[0089] The validation of interface validity can include at least one of the following: interface type validation, data type validation, data dimension validation, data format validation, interface direction validation, component category connection order validation, parameter value range validation, and logical dependency type validation.

[0090] For example, when the first component node corresponds to a situation input component and the second component node corresponds to a neural network component, it can be verified whether the situation data type, situation data fields, and situation data dimensions output by the situation input component meet the network input requirements of the neural network component. When the first component node corresponds to a neural network component and the second component node corresponds to an action output component, it can be verified whether the output dimensions of the neural network component meet the action command format, output dimensions, and parameter value range requirements of the action output component. When the connection involves a reward function component, data statistics component, or evaluation index component, it can be verified whether the fields required by the corresponding component match the output field declarations of the connected component nodes.

[0091] If the docking validity verification passes, then proceed with S504 and S505.

[0092] If the docking validity check fails, proceed to S506.

[0093] S504. Establish a directed connection from the first component node to the second component node, and generate the corresponding edge information.

[0094] The edge information includes the starting node identifier, ending node identifier, starting interface, ending interface, data flow direction, and logical dependency type. Logical dependency type can be used to characterize the functional dependencies between components, such as situation input to feature encoding, feature aggregation to action decoding, action command output to the simulation environment, and simulation feedback to the reward function.

[0095] S505. Generate a topology graph of the intelligent agent components based on the component nodes, directed connections, and the edge information corresponding to the directed connections.

[0096] S506, Output docking error message and prevent the establishment of directed connections from the first component node to the second component node.

[0097] If the docking validity verification fails, a docking error message can be output in the visual operation interface, and the establishment of a directed connection from the first component node to the second component node can be prevented.

[0098] The connection exception prompts can include the exception type, the exception component node, the exception interface, the exception cause, and correction suggestions. For example, when the output data dimension does not meet the input requirements, the prompt can be "Output dimension does not match the target component input dimension"; when the interface direction is incorrect, the prompt can be "Connection direction is incorrect, only output interfaces are allowed to connect to input interfaces".

[0099] The above combination Figure 5 This section details how the topology diagram of the intelligent agent components is generated in the embodiments of this application. The following section continues with... Figure 1 This application provides a detailed description of a visual drag-and-drop reinforcement learning agent design method based on its embodiments.

[0100] In one possible implementation, after generating the topology graph of the agent components, the topology validity of the topology graph can also be verified.

[0101] The topology validity verification can include: determining whether the agent component topology graph contains situation input component nodes, neural network component nodes, and action output component nodes used to form the operational logic of the reinforcement learning agent; determining whether a data flow path from situation data input, feature extraction and fusion to action command output is formed between situation input component nodes, neural network component nodes, and action output component nodes; determining whether there are unconnected isolated component nodes, invalid directed lines, or connection relationships that do not conform to the logical dependency type in the agent component topology graph; and determining whether the required component configuration parameters of the component nodes are configured completely.

[0102] If the topology validity check passes, the topology graph of the agent components can be marked as adaptable, allowing entry into the scenario adaptation process. If the topology validity check fails, abnormal component nodes or abnormal directed connections can be located in the agent design canvas, and a topology anomaly message can be output.

[0103] S103. Obtain the adaptation scenario corresponding to the topology diagram of the intelligent agent component, and perform scene mapping, environment interaction mapping and rule binding on the component nodes in the topology diagram of the intelligent agent component according to the adaptation scenario to obtain the scenario adaptation relationship.

[0104] Among them, the scenario is a configuration scheme used to define the training scenario, simulation operation mechanism, reward rules and data statistics rules of the reinforcement learning agent.

[0105] Specifically, the scenarios to be adapted include basic scenario information, predefined scenario data, operating mechanism, reward function adaptation rules, and data statistics rules.

[0106] The basic scenario information may include the scenario name, associated simulation components, scenario files, and decision-making methods.

[0107] Predefined scenario data can include situational data sources, environmental parameters, and scenario triggering conditions.

[0108] The operating mechanism may include the decision-making time, operating time and auxiliary rules of the agent and the environment.

[0109] The reward function adaptation rules can include scenario-specific reward and penalty quantification rules.

[0110] Data statistics rules can include data collection items during training, statistical periods, and result output formats.

[0111] Specifically, if the topology validity verification passes, the component nodes in the intelligent agent component topology graph are mapped to scenes, environment interaction and rules according to the scenario to be adapted, so as to obtain the scenario adaptation relationship.

[0112] In one possible implementation, in response to a scenario selection operation, a scenario to be adapted is obtained in association with the topology graph of the agent components.

[0113] Subsequently, the situation data source, environmental parameters, and scene triggering conditions in the predefined scene data are mapped to the corresponding situation input component nodes or simulation component nodes in the agent component topology graph to achieve scene mapping. For example, the global situation field output by the simulation environment can be mapped to the global statistical situation sub-component, the entity state field of a specified unit type can be mapped to the entity situation sub-component, and the spatial grid, terrain region, or obstacle information can be mapped to the spatial situation sub-component.

[0114] Next, based on the decision time, runtime, and auxiliary rules in the operating mechanism, the interaction cycle and rules between the design-state agent and the simulation environment corresponding to the agent component topology diagram are mapped to achieve environment interaction mapping. For example, the decision time in the scenario can be mapped to the time interval of the agent's action output, the runtime can be mapped to the duration of a single training simulation, and the auxiliary rules can be mapped to action constraints or environment adjudication constraints.

[0115] Finally, the reward function adaptation rules are bound to the reward function component nodes in the agent component topology graph, and the data statistics rules are bound to the data statistics component nodes or evaluation indicator component nodes in the agent component topology graph, thereby achieving rule binding and obtaining the desired adaptation relationship. For example, the enemy damage reward rule can be bound to the enemy damage reward component, the friendly damage penalty rule can be bound to the friendly damage reward function component, and indicators such as enemy casualties, friendly casualties, and ammunition consumption can be bound to the evaluation indicator component.

[0116] Furthermore, if no existing scenario to be adapted exists, the scenario creation operation can be responded to by configuring basic scenario information, predefined scenario data, operating mechanism, reward function adaptation rules, and data statistics rules to generate a custom scenario as the scenario to be adapted.

[0117] In this embodiment of the application, standardized adaptation between the topology diagram of intelligent agent components and the intended solution can be achieved through scene mapping, environment interaction mapping and rule binding, avoiding the need to manually write scene adaptation code.

[0118] S104. Based on the topology diagram of the agent components, component configuration parameters, and hypothetical adaptation relationships, generate the design-state agent and generate interface adaptation information for the design-state agent to interface with the agent training engine and simulation environment.

[0119] The designed agent is a reinforcement learning agent architecture model generated by the user through a visual drag-and-drop interface on an agent design canvas, where the user selects, lays out, associates, and configures parameters for components. The designed agent can include component node information, component configuration parameters, component connection relationships, network structure parameters, action output rules, reward function configuration, data statistics configuration, evaluation metric configuration, and scenario adaptation relationships.

[0120] To make it easier to understand, the following will be combined with... Figure 6 This document details how the designed intelligent agent is obtained in the embodiments of this application.

[0121] S601. Based on the logical relationships and data flow relationships in the intelligent agent component topology diagram, extract the component identifier, component configuration parameters, connection relationship information, and hypothetical adaptation relationship of the component nodes.

[0122] Specifically, it automatically identifies the topology of intelligent agent components, component configuration parameters, and hypothetical adaptation relationships. Based on the logical relationships and data flow relationships in the topology of intelligent agent components, it extracts the component identifier and connection relationship information of component nodes to obtain the component identifier, component configuration parameters, connection relationship information, and hypothetical adaptation relationships of component nodes.

[0123] Among them, the connection relationship information is used to characterize the connection relationship between component nodes, which may include directed lines, edge information, data flow direction, logical dependency type and interface mapping relationship, etc.

[0124] S602. Based on the extracted component identifiers, component configuration parameters, connection relationship information, and hypothetical adaptation relationships, generate standardized intelligent agent description information, and generate a design-state intelligent agent based on the standardized intelligent agent description information.

[0125] The standardized intelligent agent description information includes at least one of a standardized intelligent agent description file and executable construction instructions.

[0126] Standardized intelligent agent description files can be structured description files used to record component nodes, component configuration parameters, connection relationship information, and desired adaptation relationships, such as configuration files, graph structure files, or model description files.

[0127] Executable build instructions can be used to perform component instantiation, component connection, parameter writing, and interface generation based on component nodes, component configuration parameters, connection relationship information, and desired adaptation relationships.

[0128] In one possible implementation, the standardized agent description information can record the following: agent identifier, task identifier, component node list, component node type, component identifier, component configuration parameters, input / output interface definition, directed connections, edge information, scenario identifier, scene mapping relationship, environment interaction mapping relationship, reward function binding relationship, and data statistics binding relationship. The executable construction instruction can then call the component construction interface based on the above information to automatically complete component instantiation, network structure construction, action output structure construction, reward function loading, and statistical indicator binding.

[0129] S603 generates interface adaptation information for design-state intelligent agents to interface with the intelligent agent training engine and simulation environment.

[0130] The interface adaptation information may include at least one of the following: state input interface information, action output interface information, reward feedback interface information, training control interface information, and evaluation data acquisition interface information.

[0131] The state input interface information is used to input the situational data output from the simulation environment into the design-state intelligent agent.

[0132] The action output interface information is used to output the action commands generated by the design-state intelligent agent to the simulation environment.

[0133] The reward feedback interface is used to receive reward values ​​generated by the simulation environment or reward function component.

[0134] The training control interface information is used to connect to the control interfaces of the agent training engine for training start, pause, resume, and stop.

[0135] The evaluation data acquisition interface information is used to collect evaluation metrics and statistical data during the training process.

[0136] The above combination Figure 6 The previous section detailed how to obtain the designed intelligent agent; the following section will continue with... Figure 1 This application provides a detailed description of a visual drag-and-drop reinforcement learning agent design method.

[0137] S105. In response to the training start operation for the design-state agent, the design-state agent is connected to the agent training engine and simulation environment through interface adaptation information to start the reinforcement learning training of the design-state agent.

[0138] For example, when a user selects the designed agent and training environment in the visual operation interface and clicks the start training button, the training start operation for the designed agent is triggered. In response to the training start operation, the agent training engine and simulation environment are called through the interface adaptation information, and the designed agent is connected to the agent training engine and simulation environment to start the reinforcement learning training process.

[0139] During the reinforcement learning training of the designed-state agent, the simulation environment can output situational data in real time. The situational data enters the designed-state agent via the state input interface and is standardized by the situational input component. The neural network component encodes, aggregates, decodes, and evaluates the value of the data output by the situational input component. The action output component generates standardized action instructions based on the computation results of the neural network component. The simulation environment updates the simulation state based on the action instructions and generates reward values ​​according to the evaluation rules of the reward function component. The agent training engine trains and updates the agent's policy based on the state, action, and reward data.

[0140] In one possible implementation, during the reinforcement learning training of the designed agent, training-related data is collected through data statistics components and / or evaluation index components associated with the designed agent; the collected training-related data is then displayed through visualization curves or statistical panels.

[0141] The training-related data includes: training data, reward value, resource usage data, and evaluation metric data. Resource usage data may include CPU utilization, GPU utilization, video memory usage, system memory usage, and training task runtime. Evaluation metric data may include enemy casualties, friendly casualties, ammunition consumption, average reward value, and task completion rate.

[0142] In this embodiment, the collected training-related data is displayed using visual curves or statistical panels. Users can determine whether the designed agent meets the training requirements based on the training-related data displayed in the visual curves or statistical panels. If the structure or parameter configuration of the designed agent does not meet expectations, the user can return to the agent design canvas, add, delete, replace, or modify the parameters of the component nodes in the current component topology diagram, and regenerate the designed agent to continue training iterations.

[0143] To facilitate understanding, the following section uses an unmanned system simulation scenario to illustrate the application process of the visual drag-and-drop reinforcement learning agent design method provided in this application. This visual drag-and-drop reinforcement learning agent design method is applied to a visual drag-and-drop reinforcement learning agent design device.

[0144] Users need to design a reinforcement learning agent for unmanned system simulation. First, users create a new reinforcement learning agent design task in the visual interface. Based on the task requirements, users retrieve the situation input component, neural network component, action output component, reward function component, data statistics component, and evaluation metric component.

[0145] Users can select the global statistical situational awareness subcomponent, the entity situational awareness subcomponent, and the spatial situational awareness subcomponent as situational awareness input components. Among them, the global statistical situational awareness subcomponent is used to obtain global battlefield statistical information, the entity situational awareness subcomponent is used to obtain the status information of a specified unmanned system or target entity, and the spatial situational awareness subcomponent is used to obtain spatial environment information.

[0146] Users can select encoder sub-components, aggregator sub-components, decoder sub-components, and value function sub-components to construct neural network components. For example, users can select a global statistical situation encoder, an entity situation encoder, and a spatial situation encoder to encode features of different situation data, select a GRU aggregator or an LSTM aggregator to aggregate temporal features, select a classification decoder or a continuous decoder to output action decision results, and perform value evaluation through the value function sub-component.

[0147] Users can select action output components to construct the action command structure for the unmanned system. Action output components can include predicate, subject, object, and parameter sub-components. For example, the predicate sub-component is used to define action types such as "move," "attack," and "reconnaissance," the subject sub-component is used to define the unmanned system entity that performs the action, the object sub-component is used to define the action target, and the parameter sub-component is used to define action parameters, such as direction, speed, target number, or attack intensity.

[0148] After selecting the components, the user drags and drops each component onto the agent design canvas. The reinforcement learning agent design device generates corresponding component nodes in the canvas. The user establishes data flow relationships between the situation input component nodes and neural network component nodes, and between the neural network component nodes and action output component nodes, by connecting them. The reinforcement learning agent design device performs a connection validity check on each connection. If the interface type, data dimension, and data format all meet the requirements, a directed connection is established; otherwise, a connection error message is output.

[0149] After the component topology is constructed, the reinforcement learning agent design device performs topology validity verification on the agent component topology graph. When it is detected that a data flow path from situation input, feature extraction and fusion to action output has been formed in the agent component topology graph, and there are no isolated nodes, invalid connections, or missing parameters, the agent component topology graph is determined to have passed the verification.

[0150] Subsequently, the user selects or creates a simulation scenario for the unmanned system. The scenario can include simulation components, scenario files, decision-making methods, situation data sources, environmental parameters, scenario triggering conditions, agent decision-making time, simulation runtime, reward function adaptation rules, and data statistics rules. The reinforcement learning agent design device maps the situation data source to the situation input component node, maps the operating mechanism to the interaction cycle between the agent and the simulation environment, binds rules such as enemy damage rewards and friendly damage penalties to the reward function component node, and binds indicators such as enemy and friendly combat losses and ammunition consumption to the evaluation indicator component node.

[0151] After scenario adaptation is completed, the reinforcement learning agent design device generates standardized agent description information based on the agent component topology diagram, component configuration parameters, and scenario adaptation relationships, and generates the design-state agent based on the standardized agent description information. Simultaneously, the reinforcement learning agent design device generates state input interface information, action output interface information, reward feedback interface information, training control interface information, and evaluation data acquisition interface information for the design-state agent.

[0152] After the user clicks the "Start Training" button, the reinforcement learning agent design device connects the designed agent to the agent training engine and simulation environment via interface adaptation information. During training, the simulation environment outputs situational data in real time, the designed agent outputs action commands based on the situational data, the simulation environment provides reward values, and the agent training engine executes reinforcement learning training based on the interaction data. Data statistics and evaluation index components collect training-related data in real time and display the training effect through visualization curves or statistical panels. Users can adjust the agent component structure or parameters based on the training effect, thereby achieving agent training debugging and optimization.

[0153] This application provides a visual drag-and-drop reinforcement learning agent design method, comprising: responding to a component retrieval and configuration operation, determining a component to be used that matches the reinforcement learning agent design task from a pre-built component library, and setting the component configuration parameters of the component to be used; generating an agent design canvas in a visual operation interface; responding to drag-and-drop and splicing operations on the component to be used, loading the component to be used as a component node in the agent design canvas, and establishing logical relationships and data flow relationships between component nodes to generate an agent component topology graph; and obtaining the corresponding data in the agent component topology graph. The system identifies the corresponding scenario to be adapted and, based on this scenario, performs scene mapping, environment interaction mapping, and rule binding on the component nodes in the agent component topology diagram to obtain the scenario adaptation relationship. Based on the agent component topology diagram, component configuration parameters, and scenario adaptation relationship, a design-state agent is generated, and interface adaptation information for connecting to the agent training engine and simulation environment is generated for the design-state agent. In response to the training start operation for the design-state agent, the design-state agent is connected to the agent training engine and simulation environment through the interface adaptation information to initiate reinforcement learning training. This embodiment of the application, through a visual and component-based integrated agent design and training approach, lowers the technical threshold, improves the design speed and scenario adaptation speed, and achieves seamless integration of the design and training stages, thereby significantly improving the overall development and iteration efficiency of reinforcement learning agents.

[0154] Specifically, this application embodiment uses a pre-built component library to decompose the core functions of reinforcement learning agents, such as situational input, neural networks, action output, and reward functions, into reusable standardized components. Users can build agents without coding from the ground up by searching, selecting, and visually dragging and dropping components. Parameters are configured via a graphical panel, completely replacing the traditional manual coding method, significantly reducing reliance on professional algorithm engineers and lowering the technical threshold. Furthermore, the cross-task reuse of standardized components avoids the repeated development of similar modules. Combined with automatic scene mapping, environment interaction mapping, and rule binding between the scenario and the component topology graph, the process of manually writing adaptation code line by line is eliminated, significantly improving both design speed and scene adaptation speed.

[0155] Furthermore, in this embodiment, the component topology diagram, configuration parameters, and hypothetical adaptation relationships in the design phase are integrated into a design-state intelligent agent, and the interface adaptation information for connecting the training engine and simulation environment is automatically generated. Users can start training with one click, eliminating the disconnect between design and training that requires manual writing of bridging code, and achieving seamless connection between design and training.

[0156] The aforementioned closed-loop process, from component-based design and visual construction to automatic scenario adaptation and one-click training startup, allows modifications and adjustments at each stage to be completed quickly in the visual interface and instantly fed back to the training, significantly shortening the development iteration cycle and thus significantly improving the overall development iteration efficiency of reinforcement learning agents.

[0157] Device Example: The following is combined Figure 7 This application provides a detailed description of a visual drag-and-drop reinforcement learning agent design device.

[0158] like Figure 7 As shown in the figure, the visual drag-and-drop reinforcement learning agent design device provided in this application includes the following modules: The retrieval configuration module 701 is used to, in response to the component retrieval configuration operation, determine the component to be used that matches the reinforcement learning agent design task from the pre-built component library, and set the component configuration parameters of the component to be used; wherein, the pre-built component library includes standardized components obtained by encapsulating the functions of the reinforcement learning agent; The drag-and-drop splicing module 702 is used to generate an intelligent agent design canvas in a visual operation interface. In response to the drag and splicing operation of the component to be used, the component to be used is loaded as a component node in the intelligent agent design canvas, and the logical association and data flow relationship between the component nodes are established to generate an intelligent agent component topology diagram. The scenario adaptation module 703 is used to obtain the scenario to be adapted corresponding to the topology diagram of the intelligent agent component, and to perform scene mapping, environment interaction mapping and rule binding on the component nodes in the topology diagram of the intelligent agent component according to the scenario to be adapted, so as to obtain the scenario adaptation relationship. The design generation module 704 is used to generate a design-state intelligent agent based on the intelligent agent component topology diagram, the component configuration parameters and the hypothetical adaptation relationship, and to generate interface adaptation information for the design-state intelligent agent to interface with the intelligent agent training engine and simulation environment. The reinforcement training module 705 is used to respond to the training start operation for the designed agent by connecting the designed agent to the agent training engine and the simulation environment through the interface adaptation information, so as to start the reinforcement learning training of the designed agent.

[0159] In one possible implementation, the retrieval configuration module 701 is specifically used for: In response to the component retrieval operation, generate component retrieval conditions based on at least one of the component type, functional keywords, input / output interface requirements, and algorithm requirements corresponding to the reinforcement learning agent design task. Based on the component retrieval criteria, candidate components that match the reinforcement learning agent design task are filtered from the pre-built component library, and the component type, input / output interface and configurable parameters of the candidate components are displayed. In response to the component selection operation, the selected candidate component is instantiated as the component to be used; In response to the parameter configuration operation, the component configuration parameters of the component to be used are set; wherein, the component configuration parameters include one or more of the following: index name, activation status, feature information, network structure parameters, action instruction parameters, and reward quantification parameters.

[0160] In one possible implementation, the device further includes: a custom component building module for: If no candidate component matching the reinforcement learning agent design task is found in the pre-built component library based on the component retrieval conditions, the component template corresponding to the category of the custom component is invoked in response to the component creation operation. Configure the basic information, input / output interfaces, parameter items, and activation status of the custom component based on the component template; The custom component is validated for format and interface. Once the format and interface validations are passed, the custom component is added to the pre-built component library. The retrieval configuration module 701 is also used to filter candidate components that match the reinforcement learning agent design task in the pre-built component library based on the component retrieval conditions.

[0161] In one possible implementation, the drag-and-drop splicing module 702 is specifically used for: In response to a drag-and-drop operation on the component to be used, a component node corresponding to the component to be used is generated in the agent design canvas; wherein, the component node records a node identifier, a component identifier, a node position, an input interface, and an output interface; In response to the splicing operation on the component nodes, the output interface of the first component node as the starting point of the connection and the input interface of the second component node as the ending point of the connection are determined. Based on the output interface of the first component node and the input interface of the second component node, the legality of the connection between the first component node and the second component node is verified. When the connection validity verification passes, a directed connection is established from the first component node to the second component node, and corresponding edge information is generated; wherein, the edge information includes the starting node identifier, the ending node identifier, the starting interface, the ending interface, the data flow direction, and the logical dependency type; A topology graph of the intelligent agent components is generated based on the component nodes, the directed connections, and the edge information corresponding to the directed connections. The drag-and-drop splicing module 702 is also used for: When the docking validity check fails, a docking error message is output and the establishment of a directed connection from the first component node to the second component node is prevented.

[0162] In one possible implementation, the device further includes: topology validity verification, used to perform topology validity verification on the agent component topology graph; wherein, the topology validity verification includes: determining whether the agent component topology graph contains situation input component nodes, neural network component nodes, and action output component nodes for forming the operational logic of the reinforcement learning agent; determining whether a data flow path from situation data input, feature extraction and fusion to action command output is formed between the situation input component nodes, the neural network component nodes, and the action output component nodes; determining whether there are unconnected isolated component nodes, invalid directed lines, or connection relationships that do not conform to the logical dependency type in the agent component topology graph; and determining whether the required component configuration parameters of the component nodes are configured completely; If the topology validity check fails, locate the abnormal component node or abnormal directed connection in the agent design canvas and output a topology error message. The scenario adaptation module 703 is specifically used to perform scene mapping, environment interaction mapping and rule binding on the component nodes in the intelligent agent component topology diagram according to the scenario to be adapted, when the topology legality verification is passed, so as to obtain the scenario adaptation relationship.

[0163] In one possible implementation, the scenario to be adapted includes basic scenario information, predefined scenario data, operating mechanism, reward function adaptation rules, and data statistics rules; the scenario adaptation module 703 is specifically used for: In response to the scenario selection operation, obtain the scenario to be adapted associated with the topology graph of the intelligent agent component; Map the situation data source, environmental parameters and scene triggering conditions in the predefined scenario data to the corresponding situation input component node or simulation component node in the intelligent agent component topology diagram; Based on the decision time, running time and auxiliary rules in the operating mechanism, the interaction cycle and interaction rules between the design-state intelligent agent and the simulation environment corresponding to the intelligent agent component topology diagram are mapped to obtain the interaction cycle and interaction rules between the intelligent agent component topology diagram and the simulation environment. The reward function adaptation rule is bound to the reward function component node in the agent component topology graph, and the data statistics rule is bound to the data statistics component node or evaluation index component node in the agent component topology graph to obtain the desired adaptation relationship.

[0164] In one possible implementation, the generation module 704 is designed to be used specifically for: Based on the logical relationships and data flow relationships in the intelligent agent component topology diagram, extract the component identifier, component configuration parameters, connection relationship information, and hypothetical adaptation relationship of the component node; Based on the extracted component identifier, component configuration parameters, connection relationship information, and hypothetical adaptation relationship, standardized agent description information is generated, and a design-state agent is generated according to the standardized agent description information; wherein, the standardized agent description information includes at least one of: a standardized agent description file and executable construction instructions; Generate interface adaptation information for the designed intelligent agent to interface with the intelligent agent training engine and simulation environment; wherein, the interface adaptation information includes at least one of the following: state input interface information, action output interface information, reward feedback interface information, training control interface information, and evaluation data acquisition interface information.

[0165] In one possible implementation, the device further includes: a training supervision module, used for: During the reinforcement learning training process of the designed agent, training-related data are collected through data statistics components and / or evaluation index components associated with the designed agent; wherein, the training-related data includes: training data, reward value, resource consumption data and evaluation index data; The collected training-related data are displayed using visual curves or statistical panels.

[0166] This application provides a visual drag-and-drop reinforcement learning agent design device, which reduces the technical threshold, improves the design speed and scene adaptation speed, and achieves seamless connection between the design and training stages by integrating the visual and component-based agent design and training. This significantly improves the overall development and iteration efficiency of reinforcement learning agents.

[0167] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0168] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Regarding the methods disclosed in the embodiments, since they correspond to the product embodiments disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the description of the product embodiments.

[0169] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for designing a reinforcement learning agent using a visual drag-and-drop interface, characterized in that, include: In response to the component retrieval and configuration operation, a component to be used that matches the reinforcement learning agent design task is determined from the pre-built component library, and the component configuration parameters of the component to be used are set; wherein, the pre-built component library includes standardized components obtained by encapsulating the functions of the reinforcement learning agent; A smart agent design canvas is generated in the visual operation interface. In response to the drag and splice operation of the component to be used, the component to be used is loaded as a component node in the smart agent design canvas, and the logical relationship and data flow relationship between the component nodes are established to generate a smart agent component topology diagram. Obtain the adaptation scenario scheme corresponding to the topology diagram of the intelligent agent component, and perform scene mapping, environment interaction mapping and rule binding on the component nodes in the topology diagram of the intelligent agent component according to the adaptation scenario scheme to obtain the scenario adaptation relationship; Based on the topology of the agent components, the configuration parameters of the components, and the hypothetical adaptation relationship, a design-state agent is generated, and interface adaptation information for the design-state agent to interface with the agent training engine and simulation environment is generated. In response to the training start operation for the designed agent, the designed agent is connected to the agent training engine and the simulation environment through the interface adaptation information to start the reinforcement learning training of the designed agent.

2. The method according to claim 1, characterized in that, The standardized components in the pre-built component library include: simulation components, situation input components, neural network components, action output components, reward function components, data statistics components, and evaluation index components. Specifically, the simulation component configures the simulation environment mirror path and associated projects; the situation input component includes a global statistical situation sub-component, an entity situation sub-component, and a spatial situation sub-component; the neural network component includes an encoder sub-component, an aggregator sub-component, a decoder sub-component, and a value function sub-component; the action output component includes a predicate sub-component, a subject sub-component, an object sub-component, and a parameter sub-component; the reward function component defines the reward evaluation rules for agent behavior; the data statistics component provides data statistics and analysis capabilities; and the evaluation index component quantifies and evaluates the training effect and performance of the agent. Furthermore, each standardized component is configured with unique identification information and standardized input / output interfaces.

3. The method according to claim 1, characterized in that, The step of responding to the component retrieval and configuration operation involves determining the component to be used that matches the reinforcement learning agent design task from the pre-built component library, and setting the component configuration parameters of the component to be used, including: In response to the component retrieval operation, generate component retrieval conditions based on at least one of the component type, functional keywords, input / output interface requirements, and algorithm requirements corresponding to the reinforcement learning agent design task. Based on the component retrieval criteria, candidate components that match the reinforcement learning agent design task are filtered from the pre-built component library, and the component type, input / output interface and configurable parameters of the candidate components are displayed. In response to the component selection operation, the selected candidate component is instantiated as the component to be used; In response to the parameter configuration operation, the component configuration parameters of the component to be used are set; wherein, the component configuration parameters include one or more of the following: index name, activation status, feature information, network structure parameters, action command parameters, and reward quantification parameters.

4. The method according to claim 3, characterized in that, The method further includes: If no candidate component matching the reinforcement learning agent design task is found in the pre-built component library based on the component retrieval conditions, the component template corresponding to the category of the custom component is invoked in response to the component creation operation. Configure the basic information, input / output interfaces, parameter items, and activation status of the custom component based on the component template; The custom component is validated for format and interface. Once the format and interface validations are passed, the custom component is added to the pre-built component library. Based on the component retrieval criteria, candidate components matching the reinforcement learning agent design task are screened again from the pre-built component library.

5. The method according to claim 1, characterized in that, In response to drag-and-drop operations on the component to be used, the component to be used is loaded as a component node in the agent design canvas, and logical relationships and data flow relationships are established between the component nodes to generate an agent component topology graph, including: In response to a drag-and-drop operation on the component to be used, a component node corresponding to the component to be used is generated in the agent design canvas; wherein, the component node records a node identifier, a component identifier, a node position, an input interface, and an output interface; In response to the splicing operation on the component nodes, the output interface of the first component node as the starting point of the connection and the input interface of the second component node as the ending point of the connection are determined. Based on the output interface of the first component node and the input interface of the second component node, the legality of the connection between the first component node and the second component node is verified. When the connection validity verification passes, a directed connection is established from the first component node to the second component node, and corresponding edge information is generated; wherein, the edge information includes the starting node identifier, the ending node identifier, the starting interface, the ending interface, the data flow direction, and the logical dependency type; A topology graph of the intelligent agent components is generated based on the component nodes, the directed connections, and the edge information corresponding to the directed connections. The method further includes: When the docking validity check fails, a docking error message is output and the establishment of a directed connection from the first component node to the second component node is prevented.

6. The method according to claim 5, characterized in that, After generating the agent component topology graph based on the component nodes, the directed connections, and the edge information corresponding to the directed connections, the method further includes: The topology validity of the agent component topology graph is verified. This verification includes: determining whether the agent component topology graph contains situation input component nodes, neural network component nodes, and action output component nodes used to form the operational logic of the reinforcement learning agent; determining whether a data flow path from situation data input, feature extraction and fusion to action command output is formed between the situation input component nodes, the neural network component nodes, and the action output component nodes; determining whether there are unconnected isolated component nodes, invalid directed connections, or connections that do not conform to logical dependency types in the agent component topology graph; and determining whether the required component configuration parameters of the component nodes are configured completely. The step of performing scene mapping, environment interaction mapping, and rule binding on the component nodes in the intelligent agent component topology graph according to the scenario to be adapted, to obtain the scenario adaptation relationship, includes: If the topology legality verification passes, according to the proposed adaptation scenario, the component nodes in the intelligent agent component topology graph are mapped to the scene, mapped to the environment interaction, and bound to the rules to obtain the proposed adaptation relationship. The method further includes: If the topology validity check fails, locate the abnormal component node or abnormal directed connection in the agent design canvas and output a topology error message.

7. The method according to claim 1, characterized in that, The proposed adaptation scenario includes basic scenario information, predefined scenario data, operating mechanism, reward function adaptation rules, and data statistics rules. The process of obtaining the proposed adaptation scenario corresponding to the topology graph of the agent component, and performing scenario mapping, environment interaction mapping, and rule binding on the component nodes in the topology graph of the agent component based on the proposed adaptation scenario to obtain the scenario adaptation relationship, includes: In response to the scenario selection operation, obtain the scenario to be adapted associated with the topology graph of the intelligent agent component; Map the situation data source, environmental parameters and scene triggering conditions in the predefined scenario data to the corresponding situation input component node or simulation component node in the intelligent agent component topology diagram; Based on the decision time, running time and auxiliary rules in the operating mechanism, the interaction cycle and interaction rules between the design-state intelligent agent and the simulation environment corresponding to the intelligent agent component topology diagram are mapped to obtain the interaction cycle and interaction rules between the intelligent agent component topology diagram and the simulation environment. The reward function adaptation rule is bound to the reward function component node in the agent component topology graph, and the data statistics rule is bound to the data statistics component node or evaluation index component node in the agent component topology graph to obtain the desired adaptation relationship.

8. The method according to claim 1, characterized in that, Based on the agent component topology diagram, the component configuration parameters, and the hypothetical adaptation relationship, a design-state agent is generated, and interface adaptation information for the design-state agent to interface with the agent training engine and simulation environment is generated, including: Based on the logical relationships and data flow relationships in the intelligent agent component topology diagram, extract the component identifier, component configuration parameters, connection relationship information, and hypothetical adaptation relationship of the component node; Based on the extracted component identifier, component configuration parameters, connection relationship information, and hypothetical adaptation relationship, standardized agent description information is generated, and a design-state agent is generated according to the standardized agent description information; wherein, the standardized agent description information includes at least one of: a standardized agent description file and executable construction instructions; Generate interface adaptation information for the designed intelligent agent to interface with the intelligent agent training engine and simulation environment; wherein, the interface adaptation information includes at least one of the following: state input interface information, action output interface information, reward feedback interface information, training control interface information, and evaluation data acquisition interface information.

9. The method according to claim 1, characterized in that, After connecting the designed agent to the agent training engine and the simulation environment through the interface adaptation information to initiate reinforcement learning training of the designed agent, the method further includes: During the reinforcement learning training process of the designed agent, training-related data are collected through data statistics components and / or evaluation index components associated with the designed agent; wherein, the training-related data includes: training data, reward value, resource consumption data and evaluation index data; The collected training-related data are displayed using visual curves or statistical panels.

10. A visual drag-and-drop reinforcement learning agent design device, characterized in that, include: The retrieval configuration module is used to respond to the component retrieval configuration operation, determine the component to be used that matches the reinforcement learning agent design task from the pre-built component library, and set the component configuration parameters of the component to be used; wherein, the pre-built component library includes standardized components obtained by encapsulating the functions of the reinforcement learning agent; The drag-and-drop splicing module is used to generate an intelligent agent design canvas in the visual operation interface. In response to the drag and splicing operation of the component to be used, the component to be used is loaded as a component node in the intelligent agent design canvas, and the logical relationship and data flow relationship between the component nodes are established to generate an intelligent agent component topology diagram. The scenario adaptation module is used to obtain the scenario to be adapted corresponding to the topology diagram of the intelligent agent component, and to perform scene mapping, environment interaction mapping and rule binding on the component nodes in the topology diagram of the intelligent agent component according to the scenario to be adapted, so as to obtain the scenario adaptation relationship. The design generation module is used to generate a design-state intelligent agent based on the intelligent agent component topology diagram, the component configuration parameters and the hypothetical adaptation relationship, and to generate interface adaptation information for the design-state intelligent agent to interface with the intelligent agent training engine and simulation environment. The reinforcement training module is used to respond to the training start operation for the designed agent by connecting the designed agent to the agent training engine and the simulation environment through the interface adaptation information, so as to start the reinforcement learning training of the designed agent.

Images