Intelligent agent man-machine cooperation strategy training system, method, equipment and medium

Abstract

The invention discloses an agent man-machine cooperation strategy training system, method and device and a medium. The system comprises a teammate strategy population generation module, a centralized training module and a distributed optimization module. The teammate strategy population generation module is used for determining an initial strategy population according to a preset population number, and performing co-evolution on the initial strategy population to determine a target teammate strategy population; the centralized training module is used for acquiring global trajectory information according to the target teammate strategy population and performing centralized training on a global context encoder according to the global trajectory information; and the distributed optimization module is used for determining individual track information from the global track information and optimizing the local context encoder according to the individual track information. Based on the technical scheme, the cooperation effect of the intelligent agent in the complex heterogeneous environment is remarkably improved, a control strategy capable of being matched with any multi-behavior style human teammates can be achieved, and then the cooperation effect of the intelligent agent in the complex heterogeneous environment is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a training system, method, device and medium for an intelligent human-machine collaboration strategy. Background Art

[0002] Human-machine collaboration (HMC) is a collaborative work between humans and artificial intelligence that gives full play to the respective advantages of both parties and expands the application scope of artificial intelligence to a more open real world. It is of great significance to the development of my country's intelligent economy and intelligent national defense in the new era. The development of unmanned systems is particularly rapid. With their high cost-effectiveness, strong survivability and convenient intelligent use, they can replace humans to complete many repetitive, difficult and dangerous tasks. In a closed environment with clear rules, these systems can not only complete tasks excellently, but even show capabilities that surpass human experts in some areas. .

[0003] However, existing human-machine collaboration methods have made some progress in simple environments. However, these methods are mainly applied to closed environments, where the rules of the environment are clear and the boundaries are clear, while the rules of real tasks are often vague and the environment is open. In this case, the collaborative object of the agent may mutate within a single trajectory. The mutation of the trajectory will cause the agent's equilibrium point to shift, leading to the failure of collaboration. If the agent cannot quickly perceive the environmental changes caused by the mutation of teammates, the collaboration paradigm will face switching, which will eventually lead to the failure of the human-machine collaboration task. Summary of the invention

[0004] The present invention provides a training system, method, device and medium for intelligent human-machine collaboration strategy, so as to solve the problems of strategy deviation and collaboration failure caused by human strategy mutation in the process of human-machine collaboration strategy deployment in real open task scenarios, thereby enabling the human-machine collaboration method to better adapt to more open real-world application scenarios.

[0005] According to one aspect of the present invention, a training system for intelligent human-machine collaboration strategy is provided, the system comprising: a teammate strategy population generation module, a centralized training module and a distributed optimization module; wherein:

[0006] The teammate strategy population generation module is used to determine the initial strategy population according to the preset population quantity, and to perform co-evolution on the initial strategy population to determine the target teammate strategy population;

[0007] The centralized training module is used to obtain global trajectory information according to the target teammate strategy population, and to perform centralized training on the global context encoder according to the global trajectory information;

[0008] The distributed optimization module is used to determine the individual trajectory information from the global trajectory information, and optimize the local context encoder according to the individual trajectory information.

[0009] According to another aspect of the present invention, a training method for an intelligent agent human-machine collaboration strategy is provided, and the method is applied to a training system for an intelligent agent human-machine collaboration strategy, and the system comprises: a teammate strategy population generation module, a centralized training module and a distributed optimization module; wherein the method comprises:

[0010] The teammate strategy population generation module determines an initial strategy population according to a preset population quantity, and co-evolves the initial strategy population to determine a target teammate strategy population;

[0011] The centralized training module obtains global trajectory information according to the target teammate strategy population, and performs centralized training on the global context encoder according to the global trajectory information;

[0012] The distributed optimization module determines individual trajectory information from the global trajectory information and optimizes a local context encoder according to the individual trajectory information.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the intelligent agent human-computer collaboration strategy described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the training method for the intelligent agent human-computer collaboration strategy described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention determines the initial strategy population according to the preset population size through the teammate strategy population generation module, and co-evolves the initial strategy population to determine the target teammate strategy population, obtains global trajectory information according to the target teammate strategy population through the centralized training module, and centrally trains the global context encoder according to the global trajectory information, and finally determines the individual trajectory information from the global trajectory information through the distributed optimization module, and optimizes the local context encoder according to the individual trajectory information. Based on the above technical solution, the collaborative effect of the intelligent agent in a complex heterogeneous environment is significantly improved, and the control strategy that can cooperate with human teammates of any multiple behavioral styles is improved, thereby improving the collaborative effect of the intelligent agent in a complex heterogeneous environment.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a structural diagram of a training system for an intelligent agent human-machine collaboration strategy provided by an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of a method for generating heterogeneous teammates based on an evolutionary learning method provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a centralized context learning method provided by an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of an intelligent agent distributed information perception method provided by an embodiment of the present invention;

[0025] Figure 5 It is a flowchart of a training method for an intelligent agent human-machine collaboration strategy provided by an embodiment of the present invention;

[0026] Figure 6 It is a flowchart of another method for training an intelligent agent human-machine collaboration strategy provided by an embodiment of the present invention;

[0027] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 This is a structural diagram of a training system for an intelligent human-machine collaboration strategy provided by an embodiment of the present invention. Figure 1 As shown, the system includes: a teammate strategy population generation module, a centralized training module and a distributed optimization module; wherein,

[0032] The teammate strategy population generation module is used to determine the initial strategy population according to a preset population size, and to co-evolve the initial strategy population to determine the target teammate strategy population; the centralized training module is used to obtain global trajectory information according to the target teammate strategy population, and to centrally train the global context encoder according to the global trajectory information; the distributed optimization module is used to determine the individual trajectory information from the global trajectory information, and to optimize the local context encoder according to the individual trajectory information.

[0033] Among them, the global context encoder and the local context encoder are recurrent neural networks; the loss functions of the global context encoder and the local context encoder are different. The preset population size can be the number of strategies in a preset strategy population. The target teammate strategy population can be understood as a strategy population with preset conditions. Global trajectory information can be used to describe the trajectory data generated by multiple agents according to the strategies of teammates in a simulated environment. Individual trajectory information can be understood as trajectory data corresponding to the current agent, which can be split from the global trajectory information. It should be noted that in the actual deployment process, a single agent can obtain its own historical trajectory data and use the historical trajectory data as the input of the model to obtain a collaborative strategy that matches the historical trajectory information.

[0034] Specifically, a heterogeneous teammate population is generated through an evolutionary learning-based method, and the intelligent agent co-evolves with these teammates to improve its ability to adapt to different partners. Then, centralized context learning is used to achieve rapid adaptation to teammate strategy mutations through global and local context encoders, and the clustering and grouping of teammates with behavioral styles are achieved through the Chinese restaurant process. Finally, through distributed teammate situation identification and optimization, the intelligent agent can still effectively collaborate in the absence of global information in a partially observable environment. The present invention can significantly improve the collaborative effect of intelligent agents in complex heterogeneous environments, and can cooperate with control strategies of human teammates with any number of behavioral styles, providing more complete technical support for the implementation of intelligent decision-making.

[0035] Exemplarily, the teammate strategy population generation module determines the initial strategy population according to the preset population size, and the initial strategy population is co-evolved to finally determine the target teammate strategy population, and then through co-evolution, strategies that adapt to different situations and needs are generated, providing a diverse teammate strategy basis for subsequent centralized training and distributed optimization. Then, the centralized training module obtains global trajectory information according to the target teammate strategy population, and then uses the global trajectory information to conduct centralized training on the global context encoder: centralized training can ensure that the global context encoder can grasp and understand the behavior pattern and trajectory characteristics of the strategy population as a whole, and provide global perspective support for subsequent intelligent agent decisions. Finally, in order to ensure the decision-making effect under deployment, the distributed optimization module extracts the trajectory information related to the individual intelligent agent from the global trajectory information, and then optimizes the local context encoder based on these individual trajectory information. The distributed optimization enables the local context encoder to conduct refined learning and adjustment for the specific behavior and trajectory of each intelligent agent, thereby improving the adaptability and collaboration efficiency of the intelligent agent in a complex environment.

[0036] On the basis of the above technical solution, the teammate strategy population generation module includes: a teammate strategy updating unit, a teammate strategy selection unit and a target teammate strategy population determination unit; wherein the teammate strategy updating unit is used to pair the initial strategy population with the agent population to obtain a pairing result, and update the pairing result to obtain the teammate strategy to be selected; the teammate strategy selection unit is used to determine the teammate strategy population to be evaluated from the teammate strategies to be selected according to the selection weight value; the target teammate strategy population determination unit is used to determine the target teammate strategy population according to the strategy population evaluation function and the teammate strategy population to be evaluated.

[0037] Among them, the agent population can be understood as a collection of multiple agents, and the agents in the agent population work together in a specific environment to complete the task together. The pairing result can be the result of randomly combining samples in the agent population and the teammate strategy population. The teammate strategy to be selected can be the teammate strategy obtained by updating the pairing result according to a pre-set multi-agent reinforcement learning algorithm. The selection weight value can be a weight value used to select from the teammate strategy to be selected, and the weight value can be set according to demand. The strategy population evaluation function can be a pre-set function for evaluating the population value.

[0038] Specifically, in order to improve the collaboration effect in heterogeneous environments, so that the agent can better adapt to different types of partners, thereby achieving a more stable and effective collaboration strategy, by simultaneously maintaining the dual populations of agents and partners, we try to solve the problems caused by the heterogeneity and diversity of partners. Then, in the real heterogeneous collaboration task, let the agent be role 1 and the observation space be O A , the action space is The role played by humans is different from that of intelligent agents, and they also have different observation spaces and action spaces from intelligent agents, namely O H and In this formalization, the goal of the algorithm is to find the optimal agent strategy So that in collaboration with human collaborators π H The overall performance of the system during collaboration J(π A ,π H ) can be maximized. This goal can be expressed as: After the strategy population is evaluated and selected, the performance value of the teammate strategy population to be evaluated can be determined according to the objective function. The performance value can be compared with a preset threshold, and the teammate strategy population to be evaluated that meets the preset threshold can be used as the target teammate strategy population. If the current teammate strategy population to be evaluated does not meet the preset threshold, the next teammate strategy population to be evaluated is re-determined based on the current teammate strategy population to be evaluated.

[0039] On the basis of the above technical solution, the teammate strategy selection unit includes: a strategy classification subunit and a strategy selection subunit; wherein the strategy classification subunit is used to divide the teammate strategies to be selected into frozen strategy archives and clustering strategy archives, wherein the clustering strategy archives are archives obtained by clustering the teammate strategies to be selected according to the Euclidean distance; the strategy selection subunit is used to randomly select from the frozen strategy archives and the clustering strategy archives based on the selection weight value to determine the teammate strategy population to be evaluated.

[0040] The population size of the teammate strategy population to be evaluated is the same as the population size of the initial strategy population. The frozen strategy archive may be a strategy archive that has not been clustered, and correspondingly, the clustered strategy archive may be a strategy archive obtained by clustering the teammate strategies to be selected through a clustering algorithm.

[0041] Specifically, Figure 2 As shown, the technical solution of the present invention performs co-evolution on the agent and teammates respectively after random initialization. The evolution steps include iterative pairing, updating and selection. The updating stage requires training of the agent and teammates. Any multi-agent reinforcement learning algorithm can be used for optimization, and Jensen-Shannon divergence is added to the objective function to increase diversity in strategy training. The selection stage uses a clustering-based method to select with performance and diversity as evaluation criteria, and the obtained teammate strategy is used as a subset of the next generation teammate strategy population.

[0042] Based on the above technical solution, the centralized training module includes: a global trajectory determination unit and a centralized training unit; wherein the global trajectory determination unit is used to determine the target teammate strategy from the target teammate strategy population, and determine the global trajectory information according to the target teammate strategy; the centralized training unit is used to perform centralized training according to the centralized training loss function and the global trajectory information.

[0043] Among them, the target teammate strategy can be understood as a teammate strategy sampled from the target teammate strategy population.

[0044] Specifically, the global trajectory determination unit selects suitable target teammate strategies from the target teammate strategy population and determines the global trajectory information based on these strategies. For example, the global trajectory determination unit can analyze the target teammate strategy population and select representative and diverse target teammate strategies based on the performance, stability and other factors of the strategies. After selecting the target teammate strategies, the global trajectory determination unit simulates the collaboration process between these strategies and the intelligent agent, records and organizes the global trajectory information generated during the collaboration process, and then the centralized training unit is responsible for training the global context encoder based on the global trajectory information and the centralized training loss function, so as to improve the global context encoder's perception and understanding of the global environment, so that it can more accurately predict and respond to changes in teammates and human behavior. Exemplarily, a centralized training loss function is used to evaluate the performance of the global context encoder. The centralized training loss function calculates a loss value based on the global trajectory information and the output of the encoder to guide the training process of the encoder. After determining the centralized training loss function, the centralized training unit uses the global trajectory information to iteratively train the global context encoder. By continuously adjusting the parameters and structure of the encoder, it can gradually reduce the loss value and improve the perception of the global environment. During the training process, the centralized training unit will also regularly evaluate and test the performance of the global context encoder.

[0045] It should be noted that if Figure 3 As shown, by introducing the global context encoder g θ and local context encoder It is used to encode historical interaction trajectories as meaningful latent variables, thereby effectively training the agent strategy to deal with sudden changes in teammates' strategies. In the centralized training phase, a pairing-based joint strategy is used to collect trajectories in the environment. And use the global context encoder to encode the trajectory to get the latent variable And the global context encoder g θ is instantiated as a recurrent neural network that receives the trajectory as input and samples the latent variables from the output multivariate Gaussian distribution through the reparameterization technique. The global context encoding loss function used to train the encoder is defined as: in is the hidden variable learned by the context encoder in the mth environment, and M is the total number of environments. is the average shift of all past context vectors, used to replace By maximizing the correlation matrix The determinant of The diversity of , where κ is the radius of the radial basis function for calculating the vector distance.

[0046] Each batch of samples right To update: in represents gradient truncation, and η is a hyperparameter that controls the moving average. In summary, the optimization objective of this part becomes: in, represents the loss function of centralized training, represents the temporal difference loss of reinforcement learning, is the loss function defined above for optimizing the encoder, α GCE is a hyperparameter during training.

[0047] The method provided by the embodiment of the present invention can be integrated with any multi-agent reinforcement learning algorithm based on value decomposition to globally represent Can be used to assist hybrid networks, local representation and local trajectories It can also assist in calculating the local value function.

[0048] On the basis of the above technical solution, the global trajectory determination unit includes: a population clustering subunit and a global trajectory information determination subunit; wherein the population clustering subunit is used to cluster the target teammate strategy population according to the Chinese restaurant clustering algorithm to determine the target teammate strategy cluster; the global trajectory information determination subunit is used to randomly sample from the target teammate strategy cluster to determine the target teammate strategy, and determine the global trajectory information corresponding to the controllable intelligent agent according to the target teammate strategy.

[0049] The target teammate strategy cluster can be understood as a strategy set consisting of several target teammate strategies with similar characteristics generated after clustering. The controllable agent can be at least one agent that can be controlled in the current simulation environment.

[0050] Specifically, in order to cluster and group teammates based on behavioral styles, it is necessary to divide the complex teammates with diverse behaviors generated in the previous text according to high-level semantic information. This behavior takes into account that ignoring the semantic relationship between teammates will lead to a waste of too much representation information without merging teammates with similar behavior patterns, which will have a negative impact on the training of robust multi-agent collaboration strategies that can cope with various teammate strategies. In order to solve this problem, the present invention implements the clustering and division of teammates by applying the Chinese Restaurant Process (CRP), and obtains clear and identifiable teammate group boundaries from a semantic perspective, which not only retains the similarity information between teammates, but also can dynamically adjust the collaborative strategy of the agent to adapt to the behavioral changes of different teammates. Through clustering by CRP, the agent can obtain more accurate grouping information from the diverse behaviors of teammates, thereby maintaining a high degree of robustness and adaptability in multi-agent collaboration.

[0051] On the basis of the above technical solution, the distributed optimization module includes: an individual trajectory information determination unit and a distributed optimization unit; wherein the individual trajectory information determination unit is used to split the individual trajectory information corresponding to the current controllable intelligent agent from the global trajectory information; and the distributed optimization unit is used to perform distributed optimization training according to the individual trajectory information and the global trajectory information.

[0052] Specifically, Figure 4 As shown, by designing a local encoder Therefore, it can also play a similar role as the global encoder in the distributed execution stage, and the latent variables used for representation have similar semantics to the global encoder. Individual trajectory As input, the latent variable It is also sampled from the output multivariate Gaussian distribution. In order to make the distributed execution achieve the same results as the centralized training, the lower bound of the mutual information can be optimized:

[0053]

[0054] Considering the teammate representation generated by the agent Information needs and Keep consistent, so the mutual information between the two needs to be maximized, the variational distribution in the formula It is used to approximate the conditional distribution when the conditional distribution in the mutual information expression cannot be directly estimated.

[0056] Similarly, similar to the centralized training stage, the distributed stage still needs to quickly identify and adapt to the mutated teammate strategy. Therefore, the present invention also designs a local context coding loss to replace the global context coding loss when global information cannot be obtained: In addition, to obtain a descriptive representation of a specific team situation, we hope to be able to reconstruct the observations and actions taken by teammates to quickly adapt to sudden strategy changes: in Represented by ψ i Parameterized teammate modeling decoder. In summary, the optimization goal of the distributed phase becomes: Among them, α MI , α LCE and α REC is a hyperparameter during training.

[0057] The technical solution of the embodiment of the present invention determines the initial strategy population according to the preset population size through the teammate strategy population generation module, and co-evolves the initial strategy population to determine the target teammate strategy population, obtains global trajectory information according to the target teammate strategy population through the centralized training module, and centrally trains the global context encoder according to the global trajectory information, and finally determines the individual trajectory information from the global trajectory information through the distributed optimization module, and optimizes the local context encoder according to the individual trajectory information. Based on the above technical solution, the collaborative effect of the intelligent agent in a complex heterogeneous environment is significantly improved, and the control strategy that can cooperate with human teammates of any multiple behavioral styles is improved, thereby improving the collaborative effect of the intelligent agent in a complex heterogeneous environment.

[0058] Embodiment 2

[0059] Figure 5 A flowchart of a method for training an intelligent human-machine collaboration strategy provided by an embodiment of the present invention. The method is applied to a training system for an intelligent human-machine collaboration strategy, the system comprising: a teammate strategy population generation module, a centralized training module and a distributed optimization module; wherein the method comprises:

[0060] S210, the teammate strategy population generation module determines an initial strategy population according to a preset population quantity, and co-evolves the initial strategy population to determine a target teammate strategy population;

[0061] S220, the centralized training module obtains global trajectory information according to the target teammate strategy population, and performs centralized training on the global context encoder according to the global trajectory information;

[0062] S230: The distributed optimization module determines individual trajectory information from the global trajectory information, and optimizes the local context encoder according to the individual trajectory information.

[0063] On the basis of the above technical solution, it can be combined Figure 6 The technical solution of the present invention is further described as follows: Figure 6 As shown, first, a heterogeneous teammate population is generated through an evolutionary learning-based method, and the agent co-evolves with these teammates to improve its ability to adapt to different partners. Then, centralized context learning is used to achieve rapid adaptation to teammate strategy mutations through global and local context encoders, and the clustering and grouping of teammates with behavioral styles are achieved through the Chinese restaurant process. Finally, through distributed teammate situation identification and optimization, the agent can still effectively collaborate in the absence of global information in a partially observable environment. The present invention can significantly improve the collaborative effect of agents in complex heterogeneous environments, and can cooperate with control strategies of human teammates with any number of behavioral styles, providing more complete technical support for the implementation of intelligent decision-making.

[0064] The technical solution of the embodiment of the present invention determines the initial strategy population according to the preset population size through the teammate strategy population generation module, and co-evolves the initial strategy population to determine the target teammate strategy population, obtains global trajectory information according to the target teammate strategy population through the centralized training module, and centrally trains the global context encoder according to the global trajectory information, and finally determines the individual trajectory information from the global trajectory information through the distributed optimization module, and optimizes the local context encoder according to the individual trajectory information. Based on the above technical solution, the collaborative effect of the intelligent agent in a complex heterogeneous environment is significantly improved, and the control strategy that can cooperate with human teammates of any multiple behavioral styles is improved, thereby improving the collaborative effect of the intelligent agent in a complex heterogeneous environment.

[0065] Embodiment 3

[0066] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0067] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0068] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0069] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a training method for an intelligent agent human-machine collaboration strategy.

[0070] In some embodiments, the training method of the intelligent agent human-machine collaboration strategy can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the training method of the intelligent agent human-machine collaboration strategy described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the training method of the intelligent agent human-machine collaboration strategy by any other appropriate means (for example, by means of firmware).

[0071] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0072] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0073] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0074] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0075] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0076] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0077] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0078] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A training system for intelligent human-machine collaboration strategy, characterized in that: The system includes: a teammate strategy population generation module, a centralized training module and a distributed optimization module; wherein, The teammate strategy population generation module is used to determine the initial strategy population according to the preset population quantity, and to perform co-evolution on the initial strategy population to determine the target teammate strategy population; The centralized training module is used to obtain global trajectory information according to the target teammate strategy population, and to perform centralized training on the global context encoder according to the global trajectory information; The distributed optimization module is used to determine the individual trajectory information from the global trajectory information, and optimize the local context encoder according to the individual trajectory information.

2. The system according to claim 1, characterized in that The teammate strategy population generation module includes: a teammate strategy updating unit, a teammate strategy selection unit and a target teammate strategy population determination unit; wherein, The teammate strategy updating unit is used to pair the initial strategy population with the agent population to obtain a pairing result, and update the pairing result to obtain a teammate strategy to be selected; The teammate strategy selection unit is used to determine the teammate strategy population to be evaluated from the teammate strategies to be selected according to the selection weight value; The target teammate strategy population determining unit is used to determine the target teammate strategy population according to the strategy population evaluation function and the teammate strategy population to be evaluated.

3. The system according to claim 2, characterized in that The teammate strategy selection unit includes: a strategy classification subunit and a strategy selection subunit; wherein, The strategy classification subunit is used to divide the teammate strategies to be selected into a frozen strategy archive and a clustering strategy archive, wherein the clustering strategy archive is an archive obtained by clustering the teammate strategies to be selected according to the Euclidean distance; The strategy selection subunit is used to randomly select from the frozen strategy archive and the clustering strategy archive based on the selection weight value to determine the teammate strategy population to be evaluated, wherein the population size of the teammate strategy population to be evaluated is the same as the population size of the initial strategy population.

4. The system according to claim 1, characterized in that The centralized training module includes: a global trajectory determination unit and a centralized training unit; wherein The global trajectory determination unit is used to determine a target teammate strategy from the target teammate strategy population, and determine the global trajectory information according to the target teammate strategy; The centralized training unit is used to perform centralized training according to the centralized training loss function and the global trajectory information.

5. The system according to claim 4, characterized in that The global trajectory determination unit includes: a population clustering subunit and a global trajectory information determination subunit; wherein, The population clustering subunit is used to cluster the target teammate strategy population according to the Chinese restaurant clustering algorithm to determine the target teammate strategy cluster; The global trajectory information determination subunit is used to randomly sample from the target teammate strategy cluster to determine the target teammate strategy, and determine the global trajectory information corresponding to the controllable intelligent agent according to the target teammate strategy.

6. The system according to claim 1, characterized in that The distributed optimization module includes: an individual trajectory information determination unit and a distributed optimization unit; wherein, The individual trajectory information determination unit is used to separate the individual trajectory information corresponding to the current controllable intelligent agent from the global trajectory information; The distributed optimization unit is used to perform distributed optimization training according to the individual trajectory information and the global trajectory information.

7. The system according to claim 1, characterized in that The global context encoder and the local context encoder are recurrent neural networks; the loss functions of the global context encoder and the local context encoder are different.

8. A method for training an intelligent human-machine collaboration strategy, characterized in that: The method is applied to a training system for intelligent human-machine collaboration strategy, the system comprising: a teammate strategy population generation module, a centralized training module and a distributed optimization module; wherein the method comprises: The teammate strategy population generation module determines an initial strategy population according to a preset population quantity, and co-evolves the initial strategy population to determine a target teammate strategy population; The centralized training module obtains global trajectory information according to the target teammate strategy population, and performs centralized training on the global context encoder according to the global trajectory information; The distributed optimization module determines individual trajectory information from the global trajectory information and optimizes a local context encoder according to the individual trajectory information.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method for the intelligent agent human-machine collaboration strategy described in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to enable a processor to implement the training method for the intelligent agent human-computer collaboration strategy described in claim 8 when executed.

Images