A method, apparatus, device, storage medium and program product for cooperative processing

Abstract

The application provides a cooperative processing method and device, equipment, a storage medium and a program product; the application embodiment can be applied to the cooperative processing scene of cloud technology, intelligent transportation, vehicle-mounted, etc., and relates to artificial intelligence technology; the method comprises the following steps: collecting data for a displayed virtual scene to obtain scene data; based on at least one of the scene data and a received cooperation instruction, determining a corresponding to-be-executed instruction for a target virtual object in the virtual scene; wherein the to-be-executed instruction indicates an appearance area of the target virtual object and an event to be executed by the target virtual object in a future time period; in the virtual scene, outputting prompt information corresponding to the to-be-executed instruction; controlling the target virtual object to execute the to-be-executed instruction in the virtual scene to realize cooperative processing in the virtual scene. Through the application, the effect of cooperative processing can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a collaborative processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Artificial intelligence technology has been widely applied in the gaming field, such as in Go, role-playing games, and multiplayer competitive games, all of which feature characters controlled by intelligent agents, with skill levels comparable to top human players. When the intelligent agent acts as a teammate of a human player, it needs to be able to communicate effectively with the player in a virtual environment to achieve collaboration. However, in related technologies, the intelligent agent lacks effective communication with the player during human-computer collaboration, resulting in poor collaborative processing outcomes. Summary of the Invention

[0003] This application provides a collaborative processing method, apparatus, device, computer-readable storage medium, and computer program product that can improve the effectiveness of collaborative processing.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a collaborative processing method, including: Data is collected from the virtual scene being displayed to obtain scene data; the scene data describes the state of each element in the virtual scene. Based on at least one of the scene data and the received collaboration instructions, determine the corresponding instruction to be executed for the target virtual object in the virtual scene; The collaboration instruction is used to request the target virtual object in the virtual scene to collaborate, and the instruction to be executed indicates the area where the target virtual object will appear in a future time period and the event that the target virtual object needs to perform. In the virtual scene, a prompt message corresponding to the instruction to be executed is output; The target virtual object is controlled to execute the instruction to be executed in the virtual scene, so as to realize collaborative processing in the virtual scene.

[0005] This application provides a collaborative processing device, including: The data acquisition module is used to collect data from the displayed virtual scene to obtain scene data; the scene data describes the state of each element in the virtual scene. The instruction determination module is used to determine a corresponding instruction to be executed for a target virtual object in the virtual scene based on at least one of the scene data and the received collaboration instructions; wherein the collaboration instructions are used to request the target virtual object in the virtual scene to cooperate, and the instruction to be executed indicates the area where the target virtual object will appear in a future time period and the event that the target virtual object needs to perform; The prompt output module is used to output prompt information corresponding to the instruction to be executed in the virtual scene; The instruction execution module is used to control the target virtual object to execute the instruction to be executed in the virtual scene, so as to realize collaborative processing in the virtual scene.

[0006] In some embodiments of this application, the instruction determination module is further configured to determine multiple candidate instructions for the target virtual object based on the scene data and the received collaborative instructions; predict corresponding value information for each candidate instruction by combining the scene data; the value information is used to characterize the importance of the candidate instruction; and determine the instruction to be executed corresponding to the target virtual object from the multiple candidate instructions based on the value information.

[0007] In some embodiments of this application, the instruction determination module is further configured to perform feature encoding on the scene data to obtain scene encoding features; determine instruction encoding features for each candidate instruction; fuse the scene encoding features and the instruction encoding features to obtain fused encoding features for each candidate instruction; and combine the fused encoding features and the instruction encoding features to predict the value information of each candidate instruction.

[0008] In some embodiments of this application, the instruction determination module is further configured to: determine adjacent regions from the virtual scene for each candidate region corresponding to a candidate instruction; extract first matching data corresponding to the candidate region from the scene data, and perform feature encoding on the first matching data to obtain candidate region features; extract second matching data corresponding to the adjacent region from the scene data, and perform feature encoding on the second matching data to obtain adjacent region features; and determine the fusion result of the candidate region features and the adjacent region features as the instruction encoding features.

[0009] In some embodiments of this application, the instruction determination module is further configured to apply an attention mechanism to each candidate instruction based on the fused encoding features and the instruction encoding features to obtain an attention weight for each candidate instruction; and to perform value prediction on each candidate instruction based on the attention weight to obtain the value information.

[0010] In some embodiments of this application, the instruction determination module is further configured to predict an instruction probability distribution based on the scene data, and determine the instruction corresponding to the instruction probability distribution as the candidate instruction; determine the corresponding response instruction for the cooperative instruction, and determine the response instruction as the candidate instruction.

[0011] In some embodiments of this application, the prediction of the instruction probability distribution based on the scene data is achieved through an instruction encoding network; The collaborative processing device further includes: a network training module, used to acquire training scenario data and training encoding instructions; predict the probability distribution of the training scenario data using an initial encoding network to obtain a training probability distribution; determine the predicted encoding instructions corresponding to the training scenario data based on the training probability distribution; and adjust the parameters of the initial encoding network using the difference between the training encoding instructions and the predicted encoding instructions until the training termination condition is met, thereby obtaining the instruction encoding network.

[0012] In some embodiments of this application, the control of the target virtual object to execute the instruction to be executed in the virtual scene is performed by an instruction execution network; The network training module is further configured to use the acquired initial execution network to predict an action probability distribution from the predictive encoding instructions and the training scenario data; the action probability distribution characterizes the probability that the target virtual object will perform different actions based on the training scenario data; determine a first real-time stimulus based on the action probability distribution according to the training scenario data; determine a second real-time stimulus based on the action probability distribution by combining the training scenario data and the predictive encoding instructions; and adjust the parameters of the initial execution network using the first real-time stimulus and the second real-time stimulus until the training termination condition is met, thereby obtaining the instruction execution network.

[0013] In some embodiments of this application, the determination of the first real-time stimulus based on the action probability distribution according to the training scene data is achieved through a first stimulus prediction network, and the determination of the second real-time stimulus based on the action probability distribution by combining the training scene data and the predictive coding instruction is achieved through a second stimulus prediction network. The network training module is further configured to determine a first global stimulus for the action probability distribution and a second global stimulus for the predicted encoding instruction; and to adjust the parameters of the first initial stimulus network and the second initial stimulus network based on the first global stimulus and the second global stimulus to obtain the first stimulus prediction network and the second stimulus prediction network.

[0014] In some embodiments of this application, the step of combining the scene data to predict the corresponding value information for each candidate instruction is achieved through a value prediction network, wherein the predicted encoded instruction includes multiple encoded instructions. The network training module is further configured to calculate training value information for each encoded instruction; calculate the average training value using the training value information of multiple encoded instructions; determine global value information based on the training scenario data; and adjust the parameters of the initial value network according to the global value information and the average training value until the training termination condition is met, thereby obtaining the value prediction network.

[0015] In some embodiments of this application, the network training module is further configured to perform value modeling on the regional information corresponding to each encoded instruction to obtain a regional value; perform value modeling on the event information corresponding to each encoded instruction to obtain an event value; and use the regional value and the amplified event value to calculate the training value information of each encoded instruction.

[0016] This application provides an electronic device, including: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the cooperative processing method provided in the embodiments of this application.

[0017] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the cooperative processing method provided in this application.

[0018] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the cooperative processing method provided in this application.

[0019] The embodiments of this application have the following beneficial effects: The electronic device collects data from the virtual scene, clarifies the state of each element in the virtual scene, and then, based on the scene data and at least one of the collaborative instructions sent by the participating objects, determines the instructions to be executed for the target virtual object controlled by the electronic device. By outputting prompt information corresponding to the instructions to be executed, the electronic device can explicitly prompt the participating objects about the subsequent appearance area of ​​the target virtual object and the events that need to be executed, thereby realizing human-computer communication. After the human-computer communication is completed, the instructions to be executed are executed. In this way, when collaborating with the participating objects, human-computer communication can be effectively carried out, thereby improving the effect of collaborative processing. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the architecture of the collaborative processing system provided in the embodiments of this application; Figure 2 This is a schematic diagram of the server structure provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a collaborative processing method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the output prompt information provided in the embodiments of this application; Figure 5 This is another flowchart illustrating the collaborative processing method provided in the embodiments of this application; Figure 6 This is another flowchart illustrating the collaborative processing method provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the fusion of scene coding features and instruction coding features provided in an embodiment of this application; Figure 8 This is a schematic diagram of the training process of the meta-instruction encoding network provided in the embodiments of this application; Figure 9 This is a schematic diagram of the training process of the meta-instruction conditional action network provided in the embodiments of this application; Figure 10 This is a schematic diagram of the training process of the meta-instruction value selection network provided in the embodiments of this application; Figure 11 This is a schematic diagram of the structure of the meta-instruction value selection network provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0023] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0025] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0026] 1) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0027] 2) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning collectively include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0028] 3) Reinforcement Learning (RL), also known as reward learning, evaluation learning, or reinforcement learning, is one of the methods in machine learning. The basic principle of reinforcement learning is that if a certain action policy of an agent leads to positive feedback (reinforcement signal) from the environment, then the agent's tendency to produce this action policy in the future will be strengthened. The goal of the agent is to find the optimal policy in each discrete state to maximize the expected incentive.

[0029] 4) An intelligent agent refers to a computing entity that resides in a certain environment, can continuously and autonomously perform its functions, and possesses characteristics such as residency, responsiveness, sociality, and initiative. In the embodiments of this application, the electronic device used for collaborative processing can be understood as the hardware entity of the intelligent agent.

[0030] 5) A virtual scene is a virtual scene displayed (or provided) by an application when it runs on a terminal. This virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional virtual environment, or a purely fictional virtual environment. A virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene. In this embodiment, the dimension of the virtual scene is not limited. For example, a virtual scene may include the sky, land, ocean, and virtual objects, etc. The land may include environmental elements such as deserts and cities. Users and intelligent agents can control the movement of virtual objects within the virtual scene.

[0031] 6) Virtual objects: These are interactive images of people and things within a virtual scene, or movable objects within the virtual scene. These movable objects can be virtual characters, virtual animals, anime characters, etc. For example, characters, animals, plants, rocks, etc., displayed in a virtual scene. A virtual object can be a virtual representation of the user or intelligent agent within the virtual scene. A virtual scene can include multiple virtual objects, each with its own shape and volume, occupying a portion of the space within the virtual scene.

[0032] In this embodiment of the application, virtual objects in the virtual scene can be controlled by humans through operations on the client, or by a trained intelligent agent.

[0033] 7) A target virtual object refers to a virtual object in a virtual scene that is controlled by an intelligent agent (i.e., the electronic device in the embodiments of this application). For example, in a multiplayer competitive game, the character controlled by the intelligent agent is the target virtual object.

[0034] 8) Participants can be understood as humans participating in a virtual scenario, such as players in a multiplayer competitive game. Different participants control different virtual objects in the virtual scenario.

[0035] 9) Collaborative processing refers to the process by which an intelligent agent controls a target virtual object in a virtual scene to cooperate with participating objects in order to achieve a specific goal. For example, in a multiplayer competitive game, the intelligent agent controls the target virtual object and cooperates with the virtual objects controlled by the participating objects to achieve victory. In the embodiments of this application, during collaborative processing, it is necessary not only to reasonably receive instructions or prompts issued by the participating objects, but also to reasonably send instructions or prompts to the participating objects.

[0036] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0037] Artificial intelligence technology has been widely applied in the gaming field, such as in Go, role-playing games, and multiplayer competitive games, all of which feature characters controlled by intelligent agents, whose skill levels can reach those of top human players. However, for some games, simply possessing a high level of skill does not guarantee victory. For example, in multiplayer competitive games (where multiple players are divided into two opposing factions, cooperating, competing, and vying for resources in the same virtual environment, ultimately achieving victory by destroying the opponent's main base) and role-playing games, cooperation and communication among multiple players are typically required to achieve victory or complete mission objectives. Therefore, when an intelligent agent acts as a teammate of a human player, it needs to be able to communicate and cooperate with the player in a timely manner within the virtual environment.

[0038] In related technologies, intelligent agents primarily achieve collaborative learning through population training. The main idea behind this method is to allow the agent to interact with teammates exhibiting different behaviors during training, thereby improving the agent's performance in collaborative interactions with participating entities. The training process mainly consists of two phases: training a population of agents with diverse behaviors, and using this population to train the master agent.

[0039] When training a population of agents with diverse behaviors, an initial population of size N is obtained. Each agent in the population is trained independently using a self-play method, and the historical model of each agent during training is saved to represent agents with different abilities. This results in a population of agents with diverse behaviors. When training the master agent using this population, samples are continuously taken from the population, and these sampled agents become the master agent's teammates, allowing the master agent to learn to cooperate with agents exhibiting different behaviors. Thus, upon completion of training, a master agent with good generalization ability is obtained.

[0040] However, the aforementioned training methods implicitly model the collaboration between agents within the model, resulting in poor interpretability of the collaboration between agents and participating objects. This means that participating objects may not understand the agent's behavior and cannot communicate effectively. In other words, in related technologies, the lack of effective communication between agents and participating objects during human-computer collaboration leads to poor collaborative processing results.

[0041] At the same time, training a single agent usually requires a lot of computing resources, while training the master agent using a population of agents requires training each agent in the population, which also requires a huge amount of computing resources.

[0042] This application provides a collaborative processing method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the effect of collaborative processing. The following describes exemplary applications of the electronic device for collaborative processing provided in this application. The electronic device provided in this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as a server. The following describes exemplary applications when the electronic device is implemented as a server.

[0043] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of a collaborative processing system provided in an embodiment of this application. To support a collaborative processing application, in the collaborative processing system 100, terminals (terminals 400-1 and 400-2 are shown as examples) connect to server 200 (electronic devices) via network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. The collaborative processing system 100 also includes a database 500 for providing data support to server 200. Database 500 can be integrated into server 200 or operate independently of server 200. Figure 1 This illustrates the scenario where database 500 is independent of server 200.

[0044] Terminal 400-1 is used to respond to the operations of the participants on the graphical interface 410-1, generate cooperation instructions for the target virtual object in the virtual scene, and send the cooperation instructions to the server 200.

[0045] Server 200 is used to collect data from the displayed virtual scene to obtain scene data; based on the scene data and at least one of the received collaboration instructions, it determines the corresponding execution instruction for the target virtual object in the virtual scene; wherein, the collaboration instruction is used to request the target virtual object in the virtual scene to cooperate, and the execution instruction indicates the area where the target virtual object will appear in a future time period and the event to be performed by the target virtual object; in the virtual scene, it outputs prompt information corresponding to the execution instruction; and controls the target virtual object to execute the execution instruction in the virtual scene to realize collaborative processing in the virtual scene.

[0046] Terminal 400-2 is used to display prompts in the graphical interface 410-2 to prompt participants to collaborate.

[0047] The embodiments of this application can be implemented with the help of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or local area network to realize data computing, storage, processing, and sharing.

[0048] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand, flexible, and convenient use. Cloud computing technology will become a crucial support. The backend services of technical network systems require substantial computing and storage resources, which need to be realized through cloud computing.

[0049] For example, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals 400-1 and 400-2 can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, etc., but are not limited to these. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0050] See Figure 2 , Figure 2 This is a schematic diagram of the structure of a server (an embodiment of an electronic device) provided in an embodiment of this application. Figure 2The server 200 shown includes at least one processor 210, memory 250, at least one network interface 220, and a user interface 230. The various components in server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0051] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0052] User interface 230 includes one or more output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0053] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0054] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0055] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0056] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, WiFi, and Universal Serial Bus (USB). Presentation module 253 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 231 associated with user interface 230 (e.g., a display screen, a speaker, etc.). The input processing module 254 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0057] In some embodiments, the collaborative processing apparatus provided in this application can be implemented in software. Figure 2 A collaborative processing device 255 stored in memory 250 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a data acquisition module 2551, an instruction determination module 2552, a prompt output module 2553, an instruction execution module 2554, and a network training module 2555. These modules are logically connected and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0058] In other embodiments, the collaborative processing device provided in this application can be implemented in hardware. As an example, the collaborative processing device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the collaborative processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0059] In some embodiments, the terminal or server (both possible implementations of an electronic device) can implement the collaborative processing method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a game assistant APP; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0060] The embodiments of this application can be applied to collaborative processing scenarios such as cloud technology, intelligent transportation, and vehicle-mounted systems. Below, the collaborative processing method provided by the embodiments of this application will be described in conjunction with exemplary applications and implementations of the electronic devices provided in the embodiments of this application.

[0061] See Figure 3 , Figure 3 This is a flowchart illustrating a collaborative processing method provided in an embodiment of this application, which will be combined with... Figure 3 The steps shown are explained.

[0062] S101. Collect data from the virtual scene being displayed to obtain scene data.

[0063] This application's embodiments are implemented during human-computer collaboration in a virtual scene. For example, an intelligent agent and a participating object collaborate in a multiplayer competitive game to achieve victory, or they collaborate with a digital twin in the real world to accomplish a special task. It should be noted that the electronic device in this application's embodiments can be understood as the hardware entity of the intelligent agent. In this application's embodiments, after the electronic device starts the collaborative processing flow, it can collect data from the virtual scene being displayed, and determine the collected data as scene data. The scene data describes the state of various elements in the virtual scene. These elements can be any interactive object in the virtual scene, such as a defense tower, a virtual object controlled by a participating object, etc.

[0064] Scene data can include the status values ​​of various movable virtual objects, such as the health and mana of other participating objects in our faction, the health and ultimate skill cooldown of virtual objects in the opposing faction, etc. It can also include the status values ​​of virtual objects that are stationary in the virtual scene, such as the status values ​​of the opposing faction's defensive towers, the number of defensive towers in both factions, etc.

[0065] It is understandable that electronic devices can collect data by parsing the content of the displayed virtual scene. For example, they can extract various virtual objects in the virtual scene through template matching, and obtain the health value of each virtual object by identifying the quantized area (e.g., health bar) corresponding to its health value. Electronic devices can also read the runtime file corresponding to the virtual scene, locate and extract the state values ​​of each virtual object from the runtime file; however, this embodiment does not impose specific limitations on these methods.

[0066] S102. Based on at least one of the scene data and the received collaboration instructions, determine the corresponding instruction to be executed for the target virtual object in the virtual scene.

[0067] In human-machine collaboration, participating objects issue collaboration commands to a target virtual object, requesting the target virtual object controlled by the electronic device to collaborate with the virtual object controlled by the participating object. It should be noted that the sending of the collaboration command is determined by the participating object; that is, the electronic device may not receive the collaboration command. Therefore, in this embodiment, when the electronic device receives a collaboration command, it can determine the pending execution command of the target virtual object based solely on the collaboration command, or it can determine the pending execution command of the target virtual object by combining scene data and the collaboration command. In this case, the pending execution command can be an instruction responding to the collaboration command or an instruction requesting the participating object to collaborate. When the electronic device does not receive a collaboration command, it can determine the pending execution command of the target virtual object based solely on scene data. In this case, the pending execution command can be an instruction requesting the participating object to collaborate. It should be noted that the pending execution command indicates the area where the target virtual object will appear within a future time period and the event that the target virtual object will perform.

[0068] It should be noted that cooperation instructions are used to request cooperation from target virtual objects in a virtual scene. For example, a cooperation instruction can be a "request support" command issued by a participating object in the virtual scene. Based on this cooperation instruction, the electronic device can understand that the participating object's intention is to make the target virtual object reach the area where the participating object's main virtual object is located and help fight against the opposing team's virtual objects. A cooperation instruction can also be a "gather to kill monsters" command issued by a participating object in the virtual scene. Based on this command, the electronic device will understand that the participating object's intention is to make the target virtual object reach the area where the monsters are located and attack the monsters. Cooperation instructions can also be "retreat" or "opponent disappeared" commands issued by a participating object in the virtual scene. Based on this command, the electronic device can understand that the participating object's intention is to remind the target virtual object not to attack rashly, etc. Of course, cooperation instructions can also be instructions with other intentions, which are not limited in this embodiment.

[0069] Collaboration commands can be in the form of images. In this case, different graphics can be displayed in the areas corresponding to the intentions of the participating objects (e.g., displaying the graphic for gathering in the area where jungle monsters are located, and displaying the graphic for retreating in the bushes) to represent the intentions of the participating objects. Collaboration commands can also be in the form of sounds, using different statements to represent different intentions of the participating objects. For example, the statement "Request mid-lane gathering" can indicate the intention to engage in a mid-lane team fight, etc. This application does not limit the scope of the embodiments.

[0070] It can be understood that the future time period is a time period starting from the current time and whose length is less than the duration threshold. The duration threshold can be set according to the actual situation, such as 10 seconds or 30 seconds, etc., and this embodiment of the application does not make a specific limitation.

[0071] S103. In the virtual scene, output the prompt information corresponding to the instruction to be executed.

[0072] After identifying the instruction to be executed for the target virtual object, the electronic device can generate corresponding prompts for the instruction and output the prompts in the virtual scene to explicitly indicate to the participants the area where the target virtual object will appear and the event it will perform in the future. This enables communication between the participants and the device in a way that is easy for humans to understand.

[0073] It is understandable that the purpose of the prompt message matches the purpose of the instruction to be executed. That is, when the instruction to be executed is an instruction used to respond to a cooperation instruction, the prompt message is used to indicate to the participating object that the control object will cooperate with the participating object. For example, the prompt message can be a "on the way" prompt. When the instruction to be executed is an instruction used to request the participating object to cooperate, the prompt message is used to indicate to the participating object that the control object is requesting cooperation. For example, the prompt message can be a "request to gather and defeat the jungle monster" prompt, "request a mid-lane team fight", etc., and this embodiment of the application does not limit this.

[0074] The prompt message can be in graphic form, such as an arrow indicating "on the way," or it can be in voice form, such as a voice message indicating "requesting collection." This application embodiment does not limit the scope of the prompt message.

[0075] For example, Figure 4 This is a schematic diagram of the output prompt information provided in the embodiment of this application. In the virtual scene 4-11 shown in interface 4-1, prompt information 4-12 is displayed to indicate to the participants that the target virtual object will arrive at the area where the monster is located in the future time period and attack the monster.

[0076] S104. Control the target virtual object to execute the instructions to be executed in the virtual scene, so as to realize collaborative processing in the virtual scene.

[0077] Once the electronic device outputs a prompt corresponding to the instruction to be executed, it controls the target virtual object to execute the instruction within the virtual scene. This involves moving to the area indicated by the instruction and executing the event specified by it. In this way, human-computer collaboration with participating objects can be achieved.

[0078] Understandably, compared to related technologies, the lack of effective communication between the intelligent agent and the participating objects leads to poor collaborative processing results. In this embodiment, the electronic device collects data from the virtual scene to clarify the state of each element in the virtual scene. Then, based on the scene data and at least one of the collaborative instructions sent by the participating objects, it determines the instructions to be executed for the target virtual object controlled by the electronic device. By outputting prompt information corresponding to the instructions to be executed, it explicitly prompts the participating objects about the subsequent appearance area of ​​the target virtual object and the events to be executed, thereby achieving communication between humans and machines. The instructions to be executed are then executed after the communication between humans and machines is completed. In this way, when collaborating with participating objects in a virtual scene, it can effectively communicate with the participating objects, thereby improving the collaborative processing effect.

[0079] based on Figure 3 See Figure 5 , Figure 5 This is another flowchart illustrating the collaborative processing method provided in this application. In some embodiments of this application, the specific implementation process of determining the corresponding instruction to be executed for the target virtual object based on scene data and at least one received collaborative instruction, i.e., S102, may include: S1021-S1023, as follows: S1021. Based on scene data and received collaboration instructions, determine multiple candidate instructions for the target virtual object.

[0080] The electronic device first predicts commands for the target virtual object based on scene data, and then identifies the predicted commands as candidate commands. Simultaneously, the electronic device generates response commands for the received collaborative commands, and directly identifies these response commands as candidate commands. In this way, the electronic device can identify multiple candidate commands for the target virtual object.

[0081] For example, based on the health points and current location of a virtual character in the opposing faction within scene data, if an electronic device determines that a character in area A can be defeated using their ultimate skill within a future timeframe, it will generate candidate commands to travel to area A and unleash the ultimate skill within that timeframe. As another example, for a "request mid-lane team fight" coordination command, the electronic device can generate candidate commands to travel to the mid-lane and attack the opposing faction's virtual character there.

[0082] It is understandable that different candidate instructions differ in at least one of the candidate regions where the target virtual object will appear and the candidate events to be executed. For example, different candidate instructions may correspond to different candidate regions but the same candidate events, or different candidate instructions may correspond to the same candidate regions but different candidate events, or different candidate instructions may correspond to different candidate regions and different candidate events.

[0083] S1022. Based on the scenario data, predict the corresponding value information for each candidate instruction.

[0084] Value information is used to characterize the importance of candidate instructions, explaining the benefits that will be gained after the candidate instruction is executed. In other words, based on scene data, the electronic device calculates the benefits that each candidate instruction will bring after execution, such as increased virtual resources or improved attribute values, and uses this calculated information as the value information for each candidate instruction. Value information can be used to assist in selecting the final instruction to be executed for the target virtual object.

[0085] S1023. Based on value information, determine the instruction to be executed corresponding to the target virtual object from multiple candidate instructions.

[0086] After calculating the value information corresponding to each candidate instruction, the electronic device extracts the highest-value information from the value information of each candidate instruction and determines the candidate instruction corresponding to the highest-value information as the instruction to be executed for the target virtual object. In other words, the electronic device selects the candidate instruction with the highest value from multiple candidate instructions as the instruction to be executed. In this way, the electronic device can bring greater benefits to the target virtual object and even the camp to which the target virtual object belongs by subsequently executing the instruction to be executed.

[0087] based on Figure 5 See Figure 6 , Figure 6This is another flowchart illustrating the collaborative processing method provided in the embodiments of this application. In some embodiments of this application, the corresponding value information is predicted for each candidate instruction based on scene data. The specific implementation process of S1022 may include: S1022a-S1022d, as follows: S1022a. Perform feature encoding on the scene data to obtain scene encoded features.

[0088] In some embodiments, the electronic device can use scene data as input to a trained scene coding model, encode features of the scene data using the scene coding model, and use the output of the scene coding model as scene coding features. The scene coding model can be a convolutional neural network model or a belief neural network model; this embodiment does not limit the specific model.

[0089] In other embodiments, the electronic device may also match scene data with preset data for different scenes, and determine the encoding features corresponding to the preset data matched by the scene data as the scene encoding features corresponding to the scene data.

[0090] S1022b. For each candidate instruction, determine the instruction encoding characteristics.

[0091] In some embodiments, the electronic device may encode the region where the target virtual object appears and the event to be executed in each candidate instruction, and use the encoding results of the region and the encoding results of the event to fuse to obtain the instruction encoding features.

[0092] In other embodiments, the electronic device can determine adjacent regions in a virtual scene for each candidate region corresponding to a candidate instruction, and then combine the information of the adjacent regions with the information of the region where the instruction appears to encode the instruction encoding features. In this way, by extracting information from the adjacent regions of the region where each candidate instruction appears, the generalization of the value information of adjacent candidate instructions can be improved, thereby increasing the accuracy of the value information of the candidate instructions.

[0093] More specifically, the specific process of determining the instruction encoding features for each candidate instruction, i.e., S1022b, can be achieved through the following steps: For each candidate instruction corresponding to a candidate region, determine adjacent regions from the virtual scene; extract the first matching data corresponding to the candidate region from the scene data, and perform feature encoding on the first matching data to obtain the candidate region features; extract the second matching data corresponding to the adjacent region from the scene data, and perform feature encoding on the second matching data to obtain the adjacent region features; and determine the fusion result of the candidate region features and the adjacent region features as the instruction encoding features.

[0094] Here, adjacent regions are one or more regions that are adjacent to the candidate region. For example, an electronic device can uniformly divide a virtual scene into... There are several regions (n ​​can be selected based on the actual situation). (And n is an integer), the candidate region is determined by the candidate instruction. The candidate region can be any one of these regions, and the adjacent region can be the region to the left of the candidate region or the 8 surrounding regions.

[0095] After identifying adjacent regions, the electronic device extracts data corresponding to the candidate regions (i.e., the state values ​​of elements appearing in the candidate regions) from the scene data and uses this data as the first matching data. Simultaneously, it extracts data corresponding to adjacent regions (i.e., the state values ​​of elements appearing in the adjacent regions) and uses this data as the second matching data. The electronic device can use a convolutional neural network model or a fully connected network to encode the features of the first and second matching data, respectively, thereby obtaining the candidate region features and adjacent region features.

[0096] Electronic devices can fuse candidate region features and adjacent region features by splicing or by weighted summation. This application does not limit the specific methods used.

[0097] At this point, the electronic device has completed the process of determining the instruction encoding characteristics.

[0098] S1022c: The scene coding features and instruction coding features are fused to obtain the fused coding features of each candidate instruction.

[0099] In some embodiments, electronic devices can fuse scene coding features and instruction coding features through a gating mechanism. For example, Figure 7 This is a schematic diagram illustrating the fusion of scene encoding features and instruction encoding features provided in an embodiment of this application. The electronic device first extracts features from scene encoding features 7-1 through a fully connected network 7-2. The extracted features are then combined with instruction encoding features 7-4 using a gating mechanism, i.e., input into instruction gating 7-3. Simultaneously, the electronic device extracts features from instruction encoding features 7-4 through a fully connected network 7-5. The extracted features are then combined with scene encoding features 7-1 using a gating mechanism, i.e., input into scene gating 7-6. Finally, the electronic device performs a weighted summation 7-7 on the features output from instruction gating 7-3 and scene gating 7-6, resulting in the fused encoding feature 7-8.

[0100] In other embodiments, the electronic device can also directly concatenate the scene coding features and the instruction coding features, and determine the concatenation result as the fused coding feature corresponding to each candidate instruction.

[0101] S1022d combines fused coding features and instruction coding features to predict the value information of each candidate instruction.

[0102] In some embodiments, electronic devices may apply an attention mechanism to the fused coding features and instruction coding features to construct the relationship between scene data and each candidate instruction through the attention mechanism, and then perform value prediction for each candidate instruction to improve the accuracy of value information (the relationship between scene data and each candidate instruction constructed through the attention mechanism is essentially to determine the importance of each candidate instruction based on the scene data, and the benefits brought by candidate instructions with different importance after execution will necessarily be different).

[0103] More specifically, in some embodiments of this application, the specific process of predicting the value information of each candidate instruction by combining fused coding features and instruction coding features, i.e., S1022d, can be achieved through the following processing: based on fused coding features and instruction coding features, an attention mechanism is applied to each candidate instruction to obtain the attention weight of each candidate instruction; based on the attention weight, value prediction is performed on each candidate instruction to obtain the value information.

[0104] When applying an attention mechanism to each candidate instruction, the electronic device can use models such as fully connected networks and convolutional neural networks to extract features from the instruction encoding features of different candidate instructions. The extracted features and fused encoding features are used as inputs to existing attention mechanisms (such as multi-head attention mechanisms and KVQ attention mechanisms) to obtain the attention weight of each candidate instruction, thereby applying the attention mechanism to each candidate instruction. The electronic device can also perform a weighted summation of the fused encoding features and the instruction encoding features of different candidate instructions to score the instruction encoding features of different candidate instructions. The score obtained is the attention weight.

[0105] When an electronic device predicts the value information of each candidate instruction based on attention weights, it can first use attention weights to weight the candidate instructions and then input the weighted candidate instructions into the value prediction model (which can be implemented by a convolutional neural network or a fully connected network) to output the value information; or it can first input the candidate instructions into the value prediction network and then use attention weights to adjust the magnitude of the index output by the value prediction network to obtain the final value information.

[0106] In other embodiments, the electronic device may also concatenate the fused coding features and the index coding features, and use the concatenated features to match the preset features, and use the preset index corresponding to the matched preset features as the value information of the candidate instruction.

[0107] At this point, the electronic device has completed the determination of the value information of each candidate instruction, that is, the feedback obtained after the execution of each candidate instruction.

[0108] In some embodiments of this application, the specific process of determining multiple candidate instructions for a target virtual object based on scene data and received collaborative instructions, i.e., S1021, can be implemented by the following processing: predicting the instruction probability distribution based on scene data, and determining the instructions corresponding to the instruction probability distribution as candidate instructions; determining the corresponding response instructions for collaborative instructions, and determining the response instructions as candidate instructions.

[0109] The instruction probability distribution represents the likelihood of executing each instruction given scenario data. For example, given scenario data, the likelihood of executing the instructions "retreat to the high ground tower" and "go to the mid lane to participate in the team fight". Based on the instruction probability distribution, the electronic device can select the N instructions with the highest probabilities (where, Instructions (where N is a positive integer) are selected as candidate instructions.

[0110] The response command of a collaboration command is a command that controls the target virtual object to assist the virtual object controlled by the participating object. The area that the target virtual object wants to go to in the response command is the area specified by the participating object, and the event to be executed is the event specified by the participating object. For example, when the collaboration command is "Go to area A to attack jungle monsters", the area that the target virtual object wants to go to in the response command is area A, and the event to be executed is to release a skill or combo to reduce the health of the jungle monsters.

[0111] In some embodiments of this application, the prediction of the instruction probability distribution based on scene data is achieved through an instruction encoding network. In this case, before data collection for the displayed virtual scene is performed, i.e., before S101, the method may further include the following processing steps: acquiring training scene data and training encoded instructions; predicting the probability distribution of the training scene data using an initial encoding network to obtain a training probability distribution; determining the predicted encoded instructions corresponding to the training scene data based on the training probability distribution; adjusting the parameters of the initial encoding network using the difference between the training encoded instructions and the predicted encoded instructions until the training termination condition is met, thus obtaining the instruction encoding network.

[0112] It should be noted that the training scene data and training encoding instructions can be extracted from expert data corresponding to the virtual scene. For example, training scene data (which is the state of each element in the virtual scene recorded when the player interacts with the virtual scene) and training encoding instructions (which are the instructions generated when the player interacts with the virtual scene) can be extracted from the data generated by authorized players when interacting with the virtual scene. The initial encoding network can be an untrained convolutional neural network model or a recurrent neural network model (the parameters in the model are obtained through random initialization), and this embodiment of the application does not limit this.

[0113] During training, the training scenario data serves as the input to the initial encoding network, while the training encoding instructions serve as the supervision information for the initial encoding network. This enables supervised training of the initial encoding network, which involves backpropagating the differences between the training encoding instructions and the preset encoding instructions obtained based on the training scenario data into the initial encoding network to obtain the final instruction encoding network.

[0114] Understandably, electronic devices can divide supervised training data, consisting of training scenario data and training encoding instructions, into training and test sets in a 7:3 ratio. The training set is used to train the initial encoding network, and the test set is used to test the performance of the trained encoding network.

[0115] The training termination condition can be set according to the actual situation, such as the data in the training set being iterated 10,000 times, or the accuracy of the test set reaching 99%. This application embodiment does not limit this.

[0116] In some embodiments of this application, controlling the target virtual object to execute instructions in a virtual scene is performed by an instruction execution network. In this case, after determining the predicted encoded instructions corresponding to the training scene data based on the training probability distribution, the method may further include the following processing: using the acquired initial execution network, predicting the action probability distribution from the predicted encoded instructions and the training scene data; determining a first real-time stimulus based on the action probability distribution according to the training scene data; determining a second real-time stimulus based on the action probability distribution by combining the training scene data and the predicted encoded instructions; adjusting the parameters of the initial execution network using the first and second real-time stimuli until the training termination condition is met, thus obtaining the instruction execution network.

[0117] It should be noted that the action probability distribution represents the likelihood of the target virtual object performing different actions based on the training scene data. The first real-time stimulus of the action probability distribution determined by the electronic device based on the training scene data is essentially the external stimulus obtained after executing the action corresponding to the action probability distribution (this action includes reaching the area corresponding to the predictive encoded instruction and completing the event specified by the predictive encoded instruction). The second real-time stimulus determined based on the training scene data and the predictive encoded instruction is the internal stimulus obtained after executing the action corresponding to the action probability distribution. The external stimulus ensures the electronic device's operational capability for the virtual scene, while the internal stimulus ensures that the electronic device can control the target virtual object to execute and complete the generated predictive encoded instruction (i.e., ensure that the performed action matches the preset encoded instruction). Therefore, when the electronic device controls the target virtual object to move closer to the area corresponding to the predictive encoded instruction, a positive internal stimulus is generated; when the electronic device controls the target virtual object to move away from the predictive encoded instruction, a negative internal stimulus is generated.

[0118] It is understood that the electronic device can use a first stimulus prediction network for predicting external stimuli to determine a first real-time stimulus for the action probability distribution, and a second stimulus prediction network for predicting internal stimuli to determine a second real-time stimulus for the action probability distribution. Alternatively, the electronic device can read from the training scene data the changes in virtual resources obtained after executing the actions corresponding to the action probability distribution, and determine the external stimulus, i.e., the first real-time stimulus, based on these changes. The electronic device can also read from the training scene data the actual state values ​​of each element in the virtual scene after executing the actions corresponding to the action probability distribution, and generate the internal stimulus, i.e., the second real-time stimulus, based on these actual state values; however, this embodiment does not limit the scope of the invention.

[0119] In some embodiments of this application, when determining the first real-time stimulus based on the action probability distribution according to the training scene data is achieved through a first stimulus prediction network, and determining the second real-time stimulus based on the action probability distribution by combining the training scene data and the prediction encoding instruction is achieved through a second stimulus prediction network, before predicting the action probability distribution from the prediction encoding instruction and the training scene data using the obtained initial execution network, the method may further include the following processing: determining a first global stimulus for the action probability distribution, and determining a second global stimulus for the prediction encoding instruction; adjusting the parameters of the first initial stimulus network and the second initial stimulus network based on the first global stimulus and the second global stimulus to obtain the first stimulus prediction network and the second stimulus prediction network.

[0120] Specifically, the electronic device can use the average benefit from the start of the action corresponding to the action probability distribution until the end of the interaction with the virtual scene as the first global incentive, and simultaneously use the average benefit from the start of the execution of the predicted encoded instruction until the end of the interaction with the virtual scene as the second global incentive.

[0121] The electronic device can adjust the parameters of a first initial stimulus network to obtain a first stimulus prediction network with the first global stimulus as the objective, and adjust the parameters of a second initial stimulus network with the second global stimulus as the objective, to obtain a second stimulus prediction network. Alternatively, the electronic device can simultaneously adjust the parameters of the first and second initial stimulus networks with the weighted result of the first and second global stimuli as the objective, to obtain the first and second stimulus prediction networks respectively. This embodiment of the application does not limit the scope of the invention.

[0122] In some embodiments of this application, the corresponding value information is predicted for each candidate instruction by combining scene data, which is achieved through a value prediction network. Furthermore, the predicted encoded instruction includes multiple encoded instructions. In this case, after determining the predicted encoded instruction corresponding to the training scene data based on the training probability distribution, the method may further include the following processing: calculating training value information for each encoded instruction; calculating the average training value using the training value information of multiple encoded instructions; determining global value information based on the training scene data; and adjusting the parameters of the initial value network according to the global value information and the average training value until the training termination condition is met, thus obtaining the value prediction network.

[0123] Among them, training value information is used to reflect the importance of each encoded instruction. The electronic device takes the average of the training value information of multiple encoded instructions to obtain the training value mean, and determines the global value information based on the training scenario data. With the goal of making the training value mean as close as possible to the global value information and maximizing the global value information, the initial value network is trained, that is, the parameters of the initial value network are adjusted, until the training termination condition is met, and the value prediction network is obtained.

[0124] In some embodiments of this application, the specific process of calculating training value information for each encoded instruction can be implemented through the following steps: performing value modeling on the region information corresponding to each encoded instruction to obtain the region value; performing value modeling on the event information corresponding to each encoded instruction to obtain the event value; and using the region value and the amplified event value, calculating the training value information for each encoded instruction.

[0125] It should be noted that regional value can be determined during value modeling based on the number and types of elements in the virtual scene contained within the regional information corresponding to each coded instruction. For example, when the regional information contains high ground towers, bases, or a large number of virtual objects, a higher regional value (e.g., greater than a value threshold) can be generated for that regional information. Event value can be modeled based on the virtual resources obtained after executing the event. For example, when the event is defeating virtual objects of the opposing faction, or defeating virtual objects of the opposing faction consecutively, a higher event value can be generated for that event; when defeating enemy minions, a lower event value can be generated for that event.

[0126] However, in practice, event value is clearly more significant than region value (this is because region value is often determined by whether an event is accomplished; for example, going to region A to defeat a monster has no value if the monster is not defeated). Therefore, electronic devices amplify event value and use this amplified event value along with the original region value to calculate training value information. This results in more accurate training value information, which helps improve the performance of the value prediction network and ultimately enhances the accuracy of the value information for candidate instructions.

[0127] It is understood that in the embodiments of this application, the electronic device can obtain a training instruction encoding network, instruction execution network and value prediction network that can be applied to collaborative processing without training a population of intelligent agents. Thus, the embodiments of this application only need to use the time and resources to train one intelligent agent to obtain an intelligent agent with good human-machine collaboration capabilities, reducing the consumption of computing resources during intelligent agent training.

[0128] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0129] The embodiments of this application are implemented in a scenario where a server generates an intelligent agent (the intelligent agent can also run on a server, which is referred to as an electronic device). This intelligent agent can be applied in multiplayer online battle arena (MOBA) games to effectively communicate and collaborate with players (participants).

[0130] The training of the intelligent agent in this application embodiment mainly includes three parts: training of the meta-instruction encoding network (instruction encoding network), the meta-instruction conditional action network (action execution network), and the meta-instruction value selection network (value prediction network).

[0131] First, in this embodiment, observation information (training scene data) and meta-instructions (training encoding instructions) are extracted from the human expert data already authorized by the game to train the meta-instruction encoding network, so that the agent can learn the probability distribution of the meta-instructions sent by humans in the game.

[0132] Figure 8 This is a schematic diagram illustrating the training process of the meta-instruction encoding network provided in an embodiment of this application. Figure 8 In the middle, the server obtains expert data 8-1, that is... Extracting observation information from intelligent agents As input to the meta-instruction encoding network 8-2, the probability distribution of the output meta-instruction. And utilize expert data 8-1 Yuan instructions As label information, the meta-instruction encoding network 8-2 is used for supervised learning 8-3.

[0133] More specifically, in this embodiment of the application, the game map (virtual scene) can be divided into... There are 144 grids in total, each corresponding to a meta-instruction. Then, the required training data is extracted from the game's already authorized match data, specifically the observation data of the hero (target virtual object) in each game match. As input features, the meta-instructions corresponding to the grid identifier observation data where the hero will be located in the next 20 seconds are extracted. The extracted dataset was divided into training and test sets in a 7:3 ratio. The training set was used to perform supervised training of the meta-instruction encoding network, while the test set was used to test its performance. During training, an imbalanced meta-instruction sample may be observed. To address this, the focal loss function can be used to train the meta-instruction encoding network.

[0134] After the meta-instruction encoding network is trained, the server uses the trained meta-instruction encoding network to sample meta-instructions and uses the sampled meta-instructions as feature inputs to train the meta-instruction conditional action network, thereby enabling the agent to learn to execute and complete the meta-instructions.

[0135] Figure 9 This is a schematic diagram of the training process of the meta-instruction conditional action network provided in this application embodiment. First, the observation information... As input to the meta-instruction encoding network 9-1, to sample meta-instructions. and the observation information Sum instruction The common inputs are fed into the meta-instruction conditional action network 9-2, which outputs the action probability distribution of the agent. At the same time, it utilizes a network for predicting internal rewards. (Second incentive prediction network) and network for predicting external rewards (First incentive prediction network) predicts the internal and external rewards at time t, and trains the meta-instruction conditional action network 9-2 using the internal and external rewards. It should be noted that in this embodiment, meta-instructions are used... Calculate intrinsic cumulative reward (Second global incentive) and external cumulative rewards Expected sum of (first global incentives) And through reinforcement learning, that is, with Training a network to predict internal rewards with the goal of maximizing them and networks that predict external rewards .

[0136] More detailed prediction of external rewards networks It can be achieved through an external reward function To ensure the agent's micromanagement capabilities, a network predicts internal rewards. Through the intrinsic reward function This is used to incentivize agents to execute and complete meta-instructions. Specifically, during the execution of a meta-instruction, an agent receives a positive reward when it approaches the location corresponding to the meta-instruction, and a negative reward when it moves away from the location corresponding to the meta-instruction.

[0137] To improve training efficiency, in this embodiment of the application, meta-instruction input features can be added to an already trained action network to fine-tune the meta-instruction conditional action network.

[0138] After training the meta-instruction encoding network and the meta-instruction conditional action network, the agent can execute the meta-instructions issued by its teammates. However, since the meaning and value of meta-instructions issued by different teammates are different, the agent needs to model the value of the meta-instructions and select the meta-instructions with higher value to execute.

[0139] Figure 10 This is a schematic diagram illustrating the training process of the meta-instruction value selection network provided in this embodiment. First, the meta-instruction encoding network 10-1 will target different observation information, i.e. , , Sampling of meta-instructions yields meta-instructions. , , (Each encoded instruction), and all observation information and meta-instructions. (C stands for predictive coding instruction) The input is fed into the meta-instruction value selection network 10-2, which outputs the values ​​of different meta-instructions. , , (Training value information). Based on reinforcement learning, the average value for different meta-instructions is increased. With cumulative rewards close enough and To maximize the value of the meta-instruction selection network 10⁻², we train it. The cumulative reward is... Actions output by the meta-instruction conditional action network 10-3 It is confirmed that the input to the meta-instruction conditional action network 10-3 is the meta-instruction selected from the values ​​of different meta-instructions through the Softmax function. , and observation information.

[0140] Typically, the execution process of a meta-instruction can be divided into two parts: (a) navigating to the region where the meta-instruction resides (region information), and (b) executing the content of the meta-instruction (event information). For example, the meta-instruction "Gather to attack jungle monsters" includes two processes: "go to the jungle monster's location" and "defeat the jungle monster." The latter is more significant because if "defeat the jungle monster" is not achieved, then going to the "jungle monster's location" is meaningless. Therefore, it is necessary to set the value of process (b) to be greater than that of process (a). times ( ).

[0141] In this embodiment, the meta-instruction value selection network is trained using a self-play method, and three techniques are used to improve training efficiency during the training process: the optimal meta-instruction output by the meta-instruction encoding network is used as the sent meta-instruction; meta-instructions are sent at fixed intervals T, and the probability of sending a meta-instruction each time is p; and a softmax method is used to select a meta-instruction from the output of the meta-instruction value selection network for execution.

[0142] To improve the accuracy and generalization of the meta-instruction value selection network in predicting the value of meta-instructions, the following strategies are adopted in this embodiment: using a convolutional neural network to extract information from the adjacent regions of the meta-instruction (the second matching data corresponding to the adjacent regions) to improve the generalization of the value of the meta-instruction value selection network to the value of adjacent meta-instructions; using a gating mechanism to fuse the meta-instruction encoding (instruction encoding features) and the observation information encoding (scene encoding features); and introducing a target attention mechanism to improve the accuracy of the meta-instruction value selection network in predicting the value of meta-instructions in order to construct the relationship between the observation information and the meta-instruction.

[0143] For example, Figure 11This is a schematic diagram of the meta-instruction value selection network provided in this embodiment. After observation information 11-1 passes through a fully connected network 11-2, observation information encoding 11-3 is obtained. Features of adjacent meta-instructions 11-51 and 11-52 are extracted using a convolutional neural network, i.e., input into a shared convolutional neural network 11-4 for feature encoding, resulting in corresponding meta-instruction encodings 11-61 and 11-62 (collectively referred to as meta-instruction encoding 11-6). Next, a gating mechanism is applied: observation information encoding 11-3 passes through a fully connected network 11-7 and serves as the input to meta-instruction gating 11-8; meta-instruction encoding 11-6 passes through a fully connected layer 11-9 and serves as the input to observation information gating 11-10. This gating mechanism applies a gating mechanism to the observation information and meta-instructions. The output of meta-instruction gating 11-8 and meta-instruction encoding 11-6 are weighted and summed 11-11. The output of observation information gating 11-10 and observation information encoding 11-3 are weighted and summed 11-11. Then, the features obtained from the two weighted sums are fused to obtain fused encoding 11-12 (fused encoding feature). Fusion encoding 11-12 is used to calculate the mean of different meta-instructions. The input to the target attention mechanism 11-13 is used as the key in the key value 11-14 of the target attention mechanism 11-13. The input to the target attention mechanism 11-13 also includes the query 11-16 obtained after the meta-instruction encoding 11-61 and meta-instruction encoding 11-62 pass through the shared fully connected network 11-15, and then the value (value information) Q of the meta-instruction is calculated by weighted summation 11-17 with the key value 11-14. (Q and It is the value obtained by using different reinforcement learning algorithms, so that during forward prediction, the meta-instruction with the greatest value (the instruction to be executed) can be extracted from the output meta-instruction and executed.

[0144] It is understood that in the embodiments of this application, the meta-instruction conditional action network and the meta-instruction value selection network can be trained based on existing reinforcement learning algorithms, such as the PPO algorithm, DDPG algorithm, A3C algorithm, etc. The extrinsic reward function of the meta-instruction conditional action network can be calculated based on the hero's virtual resources, virtual experience, health points, etc.

[0145] It is understood that in the embodiments of this application, user information, such as collaboration instructions, training scenario data, expert data and other related data, is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0146] The following description continues to illustrate the exemplary structure of the collaborative processing device 255 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the collaborative processing device 255 of the memory 250 may include: The data acquisition module 2551 is used to acquire data from the displayed virtual scene to obtain scene data; the scene data describes the state of each element in the virtual scene. The instruction determination module 2552 is used to determine a corresponding instruction to be executed for a target virtual object in the virtual scene based on at least one of the scene data and the received collaboration instructions; wherein the collaboration instructions are used to request the target virtual object in the virtual scene to cooperate, and the instruction to be executed indicates the area where the target virtual object will appear in a future time period and the event that the target virtual object needs to perform; The prompt output module 2553 is used to output prompt information corresponding to the instruction to be executed in the virtual scene; The instruction execution module 2554 is used to control the target virtual object to execute the instruction to be executed in the virtual scene, so as to realize collaborative processing in the virtual scene.

[0147] In some embodiments of this application, the instruction determination module 2552 is further configured to determine multiple candidate instructions for the target virtual object based on the scene data and the received collaborative instructions; predict corresponding value information for each candidate instruction by combining the scene data; the value information is used to characterize the importance of the candidate instruction; and determine the instruction to be executed corresponding to the target virtual object from the multiple candidate instructions based on the value information.

[0148] In some embodiments of this application, the instruction determination module 2552 is further configured to perform feature encoding on the scene data to obtain scene encoding features; determine instruction encoding features for each candidate instruction; fuse the scene encoding features and the instruction encoding features to obtain fused encoding features for each candidate instruction; and combine the fused encoding features and the instruction encoding features to predict the value information of each candidate instruction.

[0149] In some embodiments of this application, the instruction determination module 2552 is further configured to: determine adjacent regions from the virtual scene for each candidate region corresponding to a candidate instruction; extract first matching data corresponding to the candidate region from the scene data, and perform feature encoding on the first matching data to obtain candidate region features; extract second matching data corresponding to the adjacent region from the scene data, and perform feature encoding on the second matching data to obtain adjacent region features; and determine the fusion result of the candidate region features and the adjacent region features as the instruction encoding features.

[0150] In some embodiments of this application, the instruction determination module 2552 is further configured to apply an attention mechanism to each candidate instruction based on the fused encoding features and the instruction encoding features to obtain the attention weight of each candidate instruction; and to perform value prediction on each candidate instruction based on the attention weight to obtain the value information.

[0151] In some embodiments of this application, the instruction determination module 2552 is further configured to predict an instruction probability distribution based on the scene data, and determine the instruction corresponding to the instruction probability distribution as the candidate instruction; determine the corresponding response instruction for the cooperative instruction, and determine the response instruction as the candidate instruction.

[0152] In some embodiments of this application, the prediction of the instruction probability distribution based on the scene data is achieved through an instruction encoding network; The collaborative processing device 255 further includes: a network training module 2555, used to acquire training scenario data and training encoding instructions; predict the probability distribution of the training scenario data using an initial encoding network to obtain a training probability distribution; determine the predicted encoding instructions corresponding to the training scenario data based on the training probability distribution; and adjust the parameters of the initial encoding network using the difference between the training encoding instructions and the predicted encoding instructions until the training termination condition is met, thereby obtaining the instruction encoding network.

[0153] In some embodiments of this application, the control of the target virtual object to execute the instruction to be executed in the virtual scene is performed by an instruction execution network; The network training module 2555 is further configured to use the acquired initial execution network to predict an action probability distribution from the predictive encoding instructions and the training scene data; the action probability distribution characterizes the probability that the target virtual object will perform different actions based on the training scene data; determine a first real-time stimulus based on the action probability distribution according to the training scene data; determine a second real-time stimulus based on the action probability distribution by combining the training scene data and the predictive encoding instructions; and adjust the parameters of the initial execution network using the first real-time stimulus and the second real-time stimulus until the training termination condition is met, thereby obtaining the instruction execution network.

[0154] In some embodiments of this application, the determination of the first real-time stimulus based on the action probability distribution according to the training scene data is achieved through a first stimulus prediction network, and the determination of the second real-time stimulus based on the action probability distribution by combining the training scene data and the predictive coding instruction is achieved through a second stimulus prediction network. The network training module 2555 is further configured to determine a first global stimulus for the action probability distribution and a second global stimulus for the predictive coding instruction; and to adjust the parameters of the first initial stimulus network and the second initial stimulus network based on the first global stimulus and the second global stimulus to obtain the first stimulus prediction network and the second stimulus prediction network.

[0155] In some embodiments of this application, the step of combining the scene data to predict the corresponding value information for each candidate instruction is achieved through a value prediction network, wherein the predicted encoded instruction includes multiple encoded instructions. The network training module 2555 is further configured to calculate training value information for each of the encoded instructions; calculate the average training value using the training value information of multiple encoded instructions; determine global value information based on the training scenario data; and adjust the parameters of the initial value network according to the global value information and the average training value until the training termination condition is met, thereby obtaining the value prediction network.

[0156] In some embodiments of this application, the network training module 2555 is further configured to perform value modeling on the regional information corresponding to each encoded instruction to obtain a regional value; perform value modeling on the event information corresponding to each encoded instruction to obtain an event value; and use the regional value and the amplified event value to calculate the training value information of each encoded instruction.

[0157] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device (i.e., an electronic device) reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computer device to perform the cooperative processing method described above in this application.

[0158] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the cooperative processing method provided in this application, for example... Figure 3 The collaborative processing method is shown.

[0159] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0160] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0161] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0162] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0163] In summary, through the embodiments of this application, the electronic device determines the instructions to be executed for the target virtual object controlled by the electronic device, and outputs prompt information corresponding to the instructions to be executed, thereby explicitly prompting the participating object that the target virtual object will appear in the subsequent area and that the events to be executed will be, thus realizing communication between humans and machines. After the communication between humans and machines is completed, the instructions to be executed are executed. In this way, when collaborating with the participating object in a virtual scene, human-machine communication can be effectively carried out, thereby improving the effect of collaborative processing. Only the time and resources of training one agent are needed to obtain an agent with good human-machine collaboration capabilities, reducing the consumption of computing resources during agent training.

[0164] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A collaborative processing method, characterized in that, The method includes: Data is collected from the virtual scene being displayed to obtain scene data; the scene data describes the state of each element in the virtual scene. Based on the scene data and the received collaboration instructions, multiple candidate instructions are determined for the target virtual object in the virtual scene. The multiple candidate instructions include: instructions obtained by instruction prediction for the target virtual object based on the scene data, and response instructions generated for the collaboration instructions. The scene data is feature-encoded to obtain scene-encoded features; for each candidate instruction, an instruction-encoded feature is determined; the scene-encoded features and the instruction-encoded features are fused to obtain a fused encoding feature for each candidate instruction; the fused encoding feature and the instruction-encoded feature are combined to predict the value information of each candidate instruction; the value information is used to characterize the importance of the candidate instruction. Based on the value information, the instruction to be executed corresponding to the target virtual object is determined from a plurality of candidate instructions; The collaboration instruction is used to request the target virtual object in the virtual scene to collaborate, and the instruction to be executed indicates the area where the target virtual object will appear in a future time period and the event that the target virtual object needs to perform. In the virtual scene, a prompt message corresponding to the instruction to be executed is output; The target virtual object is controlled to execute the instruction to be executed in the virtual scene, so as to realize collaborative processing in the virtual scene.

2. The method according to claim 1, characterized in that, The step of determining the instruction encoding features for each candidate instruction includes: For each candidate instruction corresponding to a candidate region, adjacent regions are determined from the virtual scene; First matching data corresponding to the candidate region is extracted from the scene data, and feature encoding is performed on the first matching data to obtain candidate region features; The second matching data corresponding to the adjacent region is extracted from the scene data, and the second matching data is feature-encoded to obtain the features of the adjacent region; The fusion result of the candidate region features and the adjacent region features is determined as the instruction encoding feature.

3. The method according to claim 1, characterized in that, The step of combining the fused coding features and the instruction coding features to predict the value information of each candidate instruction includes: Based on the fused encoding features and the instruction encoding features, an attention mechanism is applied to each candidate instruction to obtain the attention weight of each candidate instruction; Based on the attention weights, value prediction is performed on each candidate instruction to obtain the value information.

4. The method according to any one of claims 1 to 3, characterized in that, Based on the scene data and the received collaboration instructions, multiple candidate instructions are determined for the target virtual object in the virtual scene, including: Based on the scenario data, the instruction probability distribution is predicted, and the instructions corresponding to the instruction probability distribution are determined as the candidate instructions; For the cooperation instruction, a corresponding response instruction is determined, and the response instruction is identified as the candidate instruction.

5. The method according to claim 4, characterized in that, The prediction of the instruction probability distribution based on the scenario data is achieved through an instruction encoding network; Before collecting data from the displayed virtual scene to obtain scene data, the method further includes: Acquire training scenario data and training encoding instructions; The initial encoding network is used to predict the probability distribution of the training scene data to obtain the training probability distribution; Based on the training probability distribution, the predictive coding instruction corresponding to the training scenario data is determined; The parameters of the initial encoding network are adjusted by utilizing the difference between the training encoding instructions and the prediction encoding instructions until the training termination condition is met, thus obtaining the instruction encoding network.

6. The method according to claim 5, characterized in that, The control of the target virtual object to execute the instruction to be executed in the virtual scene is performed by the instruction execution network; After determining the predictive encoding instruction corresponding to the training scene data based on the training probability distribution, the method further includes: Using the acquired initial execution network, an action probability distribution is predicted from the predicted encoded instructions and the training scene data; the action probability distribution characterizes the probability that the target virtual object will perform different actions based on the training scene data. Based on the training scenario data, a first real-time stimulus is determined for the action probability distribution; By combining the training scenario data and the predictive encoding instructions, a second real-time stimulus is determined based on the action probability distribution; The parameters of the initial execution network are adjusted using the first real-time stimulus and the second real-time stimulus until the training termination condition is met, thus obtaining the instruction execution network.

7. The method according to claim 6, characterized in that, The determination of the first real-time stimulus based on the action probability distribution according to the training scene data is achieved through a first stimulus prediction network; the determination of the second real-time stimulus based on the action probability distribution by combining the training scene data and the predictive encoding instruction is achieved through a second stimulus prediction network. Before using the acquired initial execution network to predict the action probability distribution from the predictive encoded instructions and the training scene data, the method further includes: Based on the action probability distribution, a first global stimulus is determined, and based on the predictive encoded instruction, a second global stimulus is determined; Based on the first global stimulus and the second global stimulus, the parameters of the first initial stimulus network and the second initial stimulus network are adjusted respectively to obtain the first stimulus prediction network and the second stimulus prediction network.

8. The method according to claim 5, characterized in that, The prediction of the value information of each candidate instruction by combining the fused coding features and the instruction coding features is achieved through a value prediction network, and the predicted coded instructions include multiple coded instructions. After determining the predictive encoding instruction corresponding to the training scene data based on the training probability distribution, the method further includes: For each of the encoded instructions, the training value information is calculated; The average training value is calculated using the training value information of multiple encoded instructions. Global value information is determined based on the training scenario data, and the parameters of the initial value network are adjusted according to the global value information and the average training value until the training termination condition is met, thus obtaining the value prediction network.

9. The method according to claim 8, characterized in that, The calculation of training value information for each encoded instruction includes: Value modeling is performed on the regional information corresponding to each of the encoded instructions to obtain the regional value; Value modeling is performed on the event information corresponding to each encoded instruction to obtain the event value; Using the region value and the amplified event value, the training value information of each encoded instruction is calculated.

10. A collaborative processing device, characterized in that, The device includes: The data acquisition module is used to collect data from the displayed virtual scene to obtain scene data; the scene data describes the state of each element in the virtual scene. An instruction determination module is configured to determine multiple candidate instructions for a target virtual object in the virtual scene based on the scene data and received collaboration instructions. The multiple candidate instructions include: instructions obtained by instruction prediction for the target virtual object based on the scene data, and response instructions generated for the collaboration instructions. The module performs feature encoding on the scene data to obtain scene encoding features; determines instruction encoding features for each candidate instruction; fuses the scene encoding features and the instruction encoding features to obtain a fused encoding feature for each candidate instruction; combines the fused encoding feature and the instruction encoding feature to predict value information for each candidate instruction; the value information characterizes the importance of the candidate instruction; and, based on the value information, determines the execution instruction corresponding to the target virtual object from the multiple candidate instructions. The collaboration instructions request the target virtual object in the virtual scene to collaborate, and the execution instruction indicates the area where the target virtual object will appear and the event to be executed by the target virtual object within a future time period. The prompt output module is used to output prompt information corresponding to the instruction to be executed in the virtual scene; The instruction execution module is used to control the target virtual object to execute the instruction to be executed in the virtual scene, so as to realize collaborative processing in the virtual scene.

11. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the cooperative processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the cooperative processing method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, they implement the cooperative processing method according to any one of claims 1 to 9.

Images