A model training method and related device

Abstract

A model training method, relating to the field of artificial intelligence, includes: processing first data through a first reinforcement learning model to obtain a first processing result; processing the first data through a first target neural network selected from multiple first neural networks to obtain a second processing result; each first neural network being an iterative result obtained from iterative training of a first initial neural network; and updating the first reinforcement learning model based on the first processing result and the second processing result. This application utilizes historical training results of adversarial agents (adversarial agents obtained during historical iterations) to output perturbations targeting a target task, thereby obtaining more effective perturbations adapted to different scenarios for the target task, thus improving the training effect and generalization of the model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a model training method and related equipment. Background Technology

[0002] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that simulate, extend, and expand human intelligence through digital computers or machines controlled by digital computers, enabling them to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] Reinforcement learning (RL) is an important machine learning method in the field of artificial intelligence, with numerous applications in areas such as autonomous driving, intelligent robot control, and predictive analytics. Specifically, the main problem that reinforcement learning aims to solve is how intelligent devices can directly interact with their environment to learn skills for performing specific tasks, thereby maximizing long-term rewards for those tasks. In the application of reinforcement learning algorithms, interaction with an online environment is often required to obtain data and for training. A common approach is to model real-world scenarios to generate virtual simulations of the online environment. However, even slight differences between the training environment and the real-world environment to be deployed can cause the trained algorithm to fail, resulting in suboptimal performance in real-world scenarios.

[0004] The aforementioned problems can be mitigated by improving the robustness of reinforcement learning algorithms. One approach is to introduce hypothetical disturbances into a virtual environment. By training the reinforcement learning algorithm under these disturbances, its ability to cope with interference is enhanced, thereby improving its robustness and generalization. In other words, for the reinforcement learning model to be trained, an adversarial agent can be set up. The data output by this adversarial agent can be used in conjunction with the output data of the reinforcement learning model to perform the task, and the data output by the adversarial agent can serve as interference for performing the target task. However, due to the unpredictable differences between the training and deployment environments, in existing training methods, the adversarial agent can only output a specific type of disturbance (e.g., applying a force within a specific range to a joint as a disturbance in robot control). When changes in the real environment are inconsistent with the hypothetical disturbance (i.e., the disturbance output by the adversarial agent), the algorithm's performance deteriorates, resulting in poor robustness. Summary of the Invention

[0005] This application provides a model training method that can improve the training effect and generalization of the model.

[0006] In a first aspect, this application provides a model training method, the method comprising: processing first data through a first reinforcement learning model to obtain a first processing result; wherein the first data indicates the state of a target object, and the first processing result is used as control information when performing a target task on the target object; processing the first data through a first target neural network to obtain a second processing result; wherein the second processing result is used as interference information when performing the target task, the first target neural network is selected from a plurality of first neural networks, each of the first neural networks being an iterative result obtained by iteratively training a first initial neural network; performing the target task according to the first processing result and the second processing result to obtain a third processing result; and updating the first reinforcement learning model according to the third processing result to obtain an updated first reinforcement learning model.

[0007] In one possible implementation, the first reinforcement learning model can be an initialized model or the output of an iteration during model training. It should be understood that the reinforcement learning models in the embodiments of this application include, but are not limited to, deep neural networks, Bayesian neural networks, etc.

[0008] In one possible implementation, during the feedforward process of model training, a first reinforcement learning model can process the first data to obtain a first processing result. This first processing result is used as control information when performing a target task on the target object. For example, if the target task is robot posture control, the first processing result is the robot's posture control information; or, if the target task is autonomous driving of a vehicle, the first processing result is the vehicle's driving control information.

[0009] In existing implementations, an adversarial agent can be trained to output interference information, which only targets the target task. In this embodiment, on the one hand, multiple adversarial agents can be trained to output interference information, and the interference information output by different adversarial agents can target different types of interference for the target task. On the other hand, when training the adversarial agents, not only can the adversarial agents obtained in the latest iteration be used to output interference for the target task, but also the historical training results of the adversarial agents (adversarial agents obtained in the historical iteration process) can be used to output interference for the target task. This allows for more effective interference for the target task that is adapted to different scenarios, thereby improving the training effect and generalization of the model.

[0010] In one possible implementation, the first data is robot-related state information; the target task is robot posture control; and the first processing result is robot posture control information.

[0011] In one possible implementation, the robot's state information may include, but is not limited to, the robot's position, velocity, and information related to its environment (such as obstacle information). The robot's position and velocity may include information such as the state of each joint (position, angle, velocity, acceleration, etc.).

[0012] In one possible implementation, the first reinforcement learning model can obtain the robot's posture control information based on the input data. The posture control information can include the control information of each joint of the robot, and the robot's posture manipulation task can be performed based on the posture control information.

[0013] In one possible implementation, the first data is vehicle-related state information; the target task is autonomous driving of the vehicle; and the first processing result is vehicle driving control information.

[0014] In one possible implementation, vehicle-related state information may include, but is not limited to, the vehicle's position, speed, and information related to its environment (such as information about the road surface, obstacles, pedestrians, and surrounding vehicles).

[0015] In one possible implementation, the first reinforcement learning model can obtain driving control information for the vehicle based on the input data. The driving control information may include information such as the vehicle's speed, direction, and driving trajectory.

[0016] In one possible implementation, the method further includes selecting the first target neural network from the plurality of first neural networks.

[0017] In one possible implementation, the first target neural network is selected from a plurality of first neural networks based on a first selection probability corresponding to each of the plurality of first neural networks. That is, each first neural network can be configured with a probability (i.e., the first selection probability described above), and when selecting the first target neural network from the plurality of first neural networks, sampling can be performed based on the probability distribution corresponding to the plurality of first neural networks, and network selection can be performed based on the sampling results.

[0018] In one possible implementation, the processing result obtained by each first neural network is used as interference when performing the target task, and the first selection probability is positively correlated with the degree of interference of the corresponding first neural network output processing result on the target task. The first selection probability can be a trainable parameter. During the model update of the reinforcement learning model and the adversarial agent, a reward value can be obtained. This reward value can characterize both the quality of the data output by the reinforcement learning model in performing the target task and the degree of interference of the adversarial agent output. The probability distribution corresponding to the first neural network can be updated based on the reward value, so that the first selection probability is positively correlated with the degree of interference of the corresponding first neural network output processing result on the target task. Through this method, on the one hand, for adversarial agents with a large output interference range, their corresponding sampling probability is larger, making them easier to sample and increasing the degree of interference to the reinforcement learning model. On the other hand, for adversarial agents with a small output interference range, although their corresponding sampling probability is smaller, they may still be sampled, which can increase the richness of the interference to the reinforcement learning model and improve the network's generalization ability.

[0019] In one possible implementation, the probability distribution described above can be a Nash equilibrium distribution. The probability distribution can be calculated using Nash equilibrium based on the reward value obtained by performing the target task using data and interference information received during the reinforcement learning model's feedforward. The probability distribution can be updated during iterations.

[0020] This application's embodiments control the behavior space of adversarial agents, altering the intensity of their interference, thus making the reinforcement learning strategy robust to both strong and weak interference. Furthermore, by introducing a game theory optimization framework and using historical strategies to increase the diversity of adversarial agents, the reinforcement learning strategy becomes even more robust to interference from different strategies.

[0021] In one possible implementation, updating the first reinforcement learning model based on the third processing result includes:

[0022] Based on the third processing result, the reward value corresponding to the target task is obtained;

[0023] The first reinforcement learning model is updated based on the reward value;

[0024] The method further includes:

[0025] The first selection probability corresponding to the first target neural network is updated based on the reward value.

[0026] In one possible implementation, after sampling an adversarial agent for each adversarial task, the reinforcement learning policy and the updated policy of the adversarial agent can be added to the Nash equilibrium matrix, and the Nash equilibrium can be calculated to obtain the Nash equilibrium distribution of the reinforcement learning and adversarial agents. Specifically, updating the first reinforcement learning model based on the first processing result and the second processing result includes: obtaining the reward value corresponding to the target task based on the first processing result and the second processing result; updating the first reinforcement learning model based on the reward value; and then updating the first selection probability corresponding to the first target neural network based on the reward value.

[0027] In one possible implementation, to enhance the richness of the perturbations on the reinforcement learning model, multiple adversarial agents can be trained, and for each of the multiple adversarial agents, an adversarial agent that applies perturbations to the reinforcement learning model can be selected from multiple iterations during training.

[0028] In one possible implementation, the method further includes:

[0029] The first data is processed by a second target neural network to obtain a fourth processing result; wherein the fourth processing result is used as interference information when performing the target task, the second target neural network is selected from a plurality of second neural networks, each of the second neural networks being an iterative result obtained by iteratively training a second initial neural network; the first initial neural network and the second initial neural network are different;

[0030] The step of executing the target task based on the first processing result and the second processing result to obtain a third processing result includes:

[0031] Based on the first processing result, the fourth processing result, and the second processing result, the target task is executed to obtain the third processing result.

[0032] In one possible implementation, the interference types of the second processing result and the fourth processing result are different.

[0033] For example, the type of interference can be the category of interference applied when performing the target task, such as applying force, applying torque, adding obstacles, changing road conditions, changing weather, etc.

[0034] In one possible implementation, the interference objects of the second processing result and the fourth processing result are different.

[0035] For example, a robot may include multiple joints, and applying forces to different joints or different groups of joints can be considered as dealing with different objects of interference. That is, the second processing result and the fourth processing result are forces applied to different joints or different groups of joints.

[0036] In one possible implementation, the first target neural network is used to determine the second processing result from a first numerical range based on the first data, and the second target neural network is used to determine the fourth processing result from a second numerical range based on the first data, wherein the second numerical range is different from the first numerical range.

[0037] For example, both the second and fourth processing results are forces applied to the robot joints. The maximum value of the force determined by the first target neural network is A1, and the maximum value of the force determined by the second target neural network is A2. A1 and A2 are different.

[0038] In one possible implementation, during the iterative training of the adversarial agent, the reinforcement learning model participating in the current round can be selected from the historical iteration results of the reinforcement learning model. For example, this can be based on probability sampling, similar to the sampling process for the adversarial agent described in the above embodiments.

[0039] In one possible implementation, a second reinforcement learning model can be used to process the second data to obtain a fifth processing result; wherein the second reinforcement learning model is selected from a plurality of reinforcement learning models, including the updated first reinforcement learning model, and each reinforcement learning model is an iterative result obtained by iteratively training an initial reinforcement learning model; the second data indicates the state of the target object, and the fifth processing result is used as control information when performing the target task on the target object; the second data is processed by a third target neural network to obtain a sixth processing result; the third target neural network belongs to the plurality of first neural networks; the sixth processing result is used as interference information when performing the target task; the target task is performed according to the fifth processing result and the sixth processing result to obtain a seventh processing result; the third target neural network is updated according to the seventh processing result to obtain an updated third target neural network.

[0040] In one possible implementation, the second reinforcement learning model can be selected from the plurality of reinforcement learning models.

[0041] In one possible implementation, selecting the second reinforcement learning model from the plurality of reinforcement learning models includes: selecting the second reinforcement learning model from the plurality of reinforcement learning models based on a second selection probability corresponding to each reinforcement learning model among the plurality of reinforcement learning models.

[0042] In one possible implementation, the second selection probability is positively correlated with the positive performance of the corresponding reinforcement learning model's output when executing the target task. Specifically, during the model updates of the reinforcement learning model and the adversarial agent, a reward value is obtained. This reward value characterizes the quality of the reinforcement learning model's output data when executing the target task. The probability distribution corresponding to the reinforcement learning model can be updated based on the reward value, ensuring that the second selection probability is positively correlated with the positive performance of the corresponding reinforcement learning model's output when executing the target task.

[0043] In one possible implementation, historical policies of the reinforcement learning agent can be sampled from its historical policy set according to a Nash equilibrium distribution for use in updating the adversarial agent's policy. In the training environment, the selected reinforcement learning policy and the current adversarial agent policy are deployed, and sampling is performed to obtain the required training samples. These training samples are then used to train the adversarial agent's policy.

[0044] Secondly, this application provides a model training apparatus, the apparatus comprising:

[0045] A data processing module is used to process first data through a first reinforcement learning model to obtain a first processing result; wherein the first data indicates the state of a target object, and the first processing result is used as control information when performing a target task on the target object;

[0046] The first data is processed by a first target neural network to obtain a second processing result; wherein the second processing result is used as interference information when performing the target task, the first target neural network is selected from a plurality of first neural networks, and each first neural network is an iterative result obtained by iteratively training a first initial neural network;

[0047] Based on the first processing result and the second processing result, the target task is executed to obtain a third processing result;

[0048] The model update module is used to update the first reinforcement learning model according to the third processing result, so as to obtain the updated first reinforcement learning model.

[0049] In existing implementations, an adversarial agent can be trained to output interference information, which only targets the target task. In this embodiment, on the one hand, multiple adversarial agents can be trained to output interference information, and the interference information output by different adversarial agents can target different types of interference for the target task. On the other hand, when training the adversarial agents, not only can the adversarial agents obtained in the latest iteration be used to output interference for the target task, but also the historical training results of the adversarial agents (adversarial agents obtained in the historical iteration process) can be used to output interference for the target task. This allows for more effective interference for the target task that is adapted to different scenarios, thereby improving the training effect and generalization of the model.

[0050] In one possible implementation,

[0051] The target object is a robot; the target task is the robot's posture control; and the first processing result is the robot's posture control information; or...

[0052] The target object is a vehicle; the target task is the autonomous driving of the vehicle; and the first processing result is the vehicle's driving control information.

[0053] In one possible implementation, the device further includes:

[0054] A network selection module is used to select the first target neural network from the plurality of first neural networks.

[0055] In one possible implementation, the first target neural network is selected from a plurality of first neural networks based on a first selection probability corresponding to each of the plurality of first neural networks.

[0056] In one possible implementation, the processing result obtained by each first neural network from processing data is used as interference when performing the target task, and the first selection probability is positively correlated with the degree of interference of the processing result output by the corresponding first neural network on the target task.

[0057] In one possible implementation, the model update module is specifically used for:

[0058] Based on the third processing result, the reward value corresponding to the target task is obtained;

[0059] The first reinforcement learning model is updated based on the reward value;

[0060] The model update module is also used for:

[0061] The first selection probability corresponding to the first target neural network is updated based on the reward value.

[0062] In one possible implementation, the data processing module is further configured to:

[0063] The first data is processed by a second target neural network to obtain a fourth processing result; wherein the fourth processing result is used as interference information when performing the target task, the second target neural network is selected from a plurality of second neural networks, each of the second neural networks being an iterative result obtained by iteratively training a second initial neural network; the first initial neural network and the second initial neural network are different;

[0064] The data processing module is specifically used for:

[0065] Based on the first processing result, the fourth processing result, and the second processing result, the target task is executed to obtain the third processing result.

[0066] In one possible implementation,

[0067] The interference types of the second processing result and the fourth processing result are different; or,

[0068] The interference targets of the second processing result and the fourth processing result are different; or,

[0069] The first target neural network is used to determine the second processing result from a first numerical range based on the first data, and the second target neural network is used to determine the fourth processing result from a second numerical range based on the first data, wherein the second numerical range is different from the first numerical range.

[0070] In one possible implementation, the data processing module is further configured to:

[0071] The second reinforcement learning model processes the second data to obtain a fifth processing result; wherein the second reinforcement learning model is selected from multiple reinforcement learning models, including the updated first reinforcement learning model, and each reinforcement learning model is an iterative result obtained by iteratively training an initial reinforcement learning model; the second data indicates the state of the target object, and the fifth processing result is used as control information when performing the target task on the target object;

[0072] The second data is processed by a third target neural network to obtain a sixth processing result; the third target neural network belongs to the plurality of first neural networks; the sixth processing result is used as interference information when performing the target task.

[0073] Based on the fifth and sixth processing results, the target task is executed to obtain the seventh processing result;

[0074] The model update module is also used for:

[0075] Based on the seventh processing result, the third target neural network is updated to obtain the updated third target neural network.

[0076] In one possible implementation, the network selection module is further configured to:

[0077] The second reinforcement learning model is selected from the plurality of reinforcement learning models.

[0078] In one possible implementation, the network selection module is specifically used for:

[0079] The second reinforcement learning model is selected from the plurality of reinforcement learning models based on the second selection probability corresponding to each reinforcement learning model.

[0080] Thirdly, this application provides a data processing method, including:

[0081] Acquire first data, which indicates the state of the target object;

[0082] The first data is processed using a first reinforcement learning model to obtain a first processing result; the first processing result is used as control information for the target object; wherein...

[0083] The first reinforcement learning model is updated with a reward value during one iteration of training. The reward value is obtained by applying interference information when executing the target task based on the control information output by the feedforward process of the first reinforcement learning model. The interference information is obtained through the feedforward process of the target neural network. The target neural network is selected from multiple neural networks, and each neural network is an iterative result obtained by iteratively training the initial neural network.

[0084] Based on the first processing result, the target task is performed on the target object.

[0085] In one possible implementation,

[0086] The target object is a robot; the target task is the robot's posture control; and the first processing result is the robot's posture control information; or...

[0087] The target object is a vehicle; the target task is the autonomous driving of the vehicle; and the first processing result is the vehicle's driving control information.

[0088] In one possible implementation, the target neural network is selected from a plurality of first neural networks based on a first selection probability corresponding to each of the plurality of neural networks.

[0089] In one possible implementation, the processing result obtained by each neural network from processing data is used as interference when performing the target task, and the first selection probability is positively correlated with the degree of interference of the processing result of the corresponding neural network output on the target task.

[0090] Fourthly, embodiments of this application provide a model training apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect above and any of its optional methods.

[0091] Fifthly, embodiments of this application provide a data processing apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the third aspect above and any of its optional methods.

[0092] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any optional methods thereof, or the methods described in the third aspect and any optional methods thereof.

[0093] In a seventh aspect, embodiments of this application provide a computer program product including instructions that, when run on a computer, cause the computer to perform the first aspect and any optional method thereof, or the third aspect and any optional method thereof.

[0094] Eighthly, this application provides a chip system including a processor for supporting a model training device in implementing some or all of the functions involved in the foregoing aspects, such as transmitting or processing data or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices.

[0095] This application provides a model training method, comprising: processing first data through a first reinforcement learning model to obtain a first processing result; wherein the first data indicates the state of a target object, and the first processing result is used as control information when performing a target task on the target object; processing the first data through a first target neural network to obtain a second processing result; wherein the second processing result is used as interference information when performing the target task, the first target neural network is selected from a plurality of first neural networks, each of the first neural networks being an iterative result obtained by iteratively training a first initial neural network; performing the target task according to the first processing result and the second processing result to obtain a third processing result; and updating the first reinforcement learning model according to the third processing result to obtain an updated first reinforcement learning model. Through the above method, when training an adversarial agent, not only can the adversarial agent obtained in the latest iteration be used to output interference for the target task, but also the historical training results of the adversarial agent (adversarial agents obtained in the historical iteration process) can be used to output interference for the target task, thereby obtaining more effective interference adapted to different scenarios for the target task, thereby improving the training effect and generalization of the model. Attached Figure Description

[0096] Figure 1 This is a schematic diagram of an application architecture;

[0097] Figure 2 This is a schematic diagram of an application architecture;

[0098] Figure 3 This is a schematic diagram of an application architecture;

[0099] Figure 4 This is an illustration of an embodiment of a model training method provided in this application.

[0100] Figure 5 A schematic diagram of a software architecture provided for an embodiment of this application;

[0101] Figure 6 This is an illustration of an embodiment of a model training method provided in this application.

[0102] Figure 7 This application provides an embodiment of a model training apparatus.

[0103] Figure 8 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0104] Figure 9 This is a schematic diagram of a server structure provided in an embodiment of this application;

[0105] Figure 10 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0106] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0107] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0108] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0109] The terms “substantially,” “about,” and similar terms used herein are used as approximations rather than as terms of degree, and are intended to take into account the inherent biases of measurements or calculations known to those skilled in the art. Furthermore, the use of “may” in describing embodiments of the invention refers to “one or more possible embodiments.” The terms “use,” “using,” and “used” used herein are to be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Additionally, the term “exemplary” is intended to refer to an instance or illustration.

[0110] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0111] (1) Infrastructure

[0112] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0113] (2) Data

[0114] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, as well as IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0115] (3) Data processing

[0116] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0117] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0118] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0119] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0120] (4) General ability

[0121] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0122] (5) Smart products and industry applications

[0123] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0124] With the development of artificial intelligence, many tasks that previously required human intervention are gradually being replaced by smart terminals. These smart terminals need to be equipped with the skills required to complete these tasks, as well as neural networks specific to those tasks, to enable them to perform specific tasks. Specifically, this can be applied to mobile smart terminals. For example, in the field of autonomous driving, driving operations that were originally performed by humans can be performed by smart cars. Therefore, smart cars need to be equipped with a large number of driving skills and neural networks specific to those skills. As another example, in the freight industry, handling operations that were originally performed by humans can be performed by handling robots. Therefore, handling robots need to be equipped with a large number of handling skills and neural networks specific to those skills. It can also be applied to smart terminals that do not have mobile operation capabilities. For example, on a parts processing assembly line, the parts grasping operation that was originally done by humans can be done by a smart robotic arm. In this case, the smart robotic arm needs to be equipped with grasping skills and neural networks for grasping skills. Different grasping skills can have different grasping angles and displacements of the smart robotic arm. As another example, in the field of automatic cooking, the cooking operation that was originally done by humans can be done by a smart robotic arm. In this case, the smart robotic arm needs to be equipped with raw material grasping skills, stir-frying skills, and other cooking skills, as well as neural networks for cooking skills, etc. Other application scenarios will not be exhaustively listed here.

[0125] To better understand the solutions of the embodiments of this application, the following will first combine... Figure 2 and Figure 3 A brief introduction to the possible implementation architectures of the embodiments of this application is provided.

[0126] Figure 2This is a schematic diagram of a computing system for performing model training in an embodiment of this application. The computing system includes a terminal device 102 (exemplary, but not necessarily including the terminal device 102) and a server 130 (also referred to as a central node) coupled via a network communication. The terminal device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0127] Terminal device 102 may include processor 112 and memory 114. Processor 112 may be any suitable processing device (e.g., processor core, microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), controller, microcontroller, etc.). Memory 114 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). Memory 114 may store data 116 and instructions 118 executed by processor 112 to cause terminal device 102 to perform operations.

[0128] In some implementations, memory 114 may store one or more models 120. For example, model 120 may be or may additionally include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks.

[0129] In some implementations, one or more models 120 may be received from server 130 via network 180, stored in memory 114, and then used or otherwise implemented by one or more processors 112.

[0130] Terminal device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). Touch-sensitive components can be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices from which the user can provide input.

[0131] Terminal device 102 may also include a communication interface 123. Terminal device 102 can communicate with server 130 through communication interface 123. Server 130 may include a communication interface 133. Terminal device 102 can communicate with server 130 through communication interface 123 and communication interface 133, thereby realizing data interaction between terminal device 102 and server 130.

[0132] Server 130 may include processor 132 and memory 134. Processor 132 may be any suitable processing device (e.g., processor core, microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), controller, microcontroller, etc.). Memory 134 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). Memory 134 may store data 136 and instructions 138 executed by processor 132 to cause server 130 to perform operations.

[0133] As described above, memory 134 can store one or more machine learning models 140. For example, model 140 can be, or may additionally include, various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0134] It should be understood that the model training method in this application involves AI-related computations. When performing AI computations, the instruction execution architecture of the terminal device and the server is not limited to... Figure 2 The processor and memory architecture shown is illustrated below. Figure 3The system architecture provided in the embodiments of this application will be described in detail.

[0135] Figure 3 This is a schematic diagram of the system architecture provided for an embodiment of this application. Figure 3 As shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0136] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, while the preprocessing modules 513 and 514 are optional.

[0137] The data acquisition device 560 is used to collect training samples. The training samples can be first data, second data, etc., where the first data and second data can be state information related to the target object (e.g., robot, vehicle, etc.), vehicle-related state information, etc. After collecting the training samples, the data acquisition device 560 stores these training samples in the database 530.

[0138] The training device 520 can maintain training samples in the database 530 and obtain the target model / rule 501 from the neural network to be trained (such as the reinforcement learning model and the target neural network in the embodiments of this application, wherein the target neural network is used as the adversarial agent of the reinforcement learning model).

[0139] It should be noted that in practical applications, the training samples maintained in database 530 may not all come from the data acquisition device 560; they may also be received from other devices. Furthermore, it should be noted that training device 520 may not necessarily train the target model / rule 501 entirely based on the training samples maintained in database 530; it may also obtain training samples from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0140] The target model / rule 501 trained using training device 520 can be applied to different systems or devices, such as... Figure 3 The execution device 510 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server, etc.

[0141] Among them, the target model / rule 501 can be used to achieve target tasks, such as driving control in autonomous driving and posture control on robots.

[0142] Specifically, the training device 520 can transfer the trained model to the execution device 510. The execution device 510 can be the target object mentioned above.

[0143] exist Figure 3 In the execution device 510, an input / output (I / O) interface 512 is configured for data interaction with external devices. Users can input data to the I / O interface 512 through the client device 540, or the execution device 510 can automatically collect input data.

[0144] Preprocessing modules 513 and 514 are used to preprocess the input data received from the I / O interface 512. It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can be used directly to process the input data.

[0145] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0146] Finally, the I / O interface 512 provides the processing result to the client device 540, which then provides it to the user or performs control operations based on the processing result.

[0147] exist Figure 3In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various forms such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0148] It is worth noting that, Figure 3 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3 In this context, the data storage system 550 is an external storage device relative to the execution device 510. However, in other cases, the data storage system 550 may also be placed within the execution device 510. It should be understood that the aforementioned execution device 510 may be deployed within the client device 540.

[0149] From the training side of the model:

[0150] In this embodiment of the application, the training device 520 can access the memory ( Figure 3 (Not shown in the diagram, but can be integrated into the training device 520 or deployed separately from the training device 520) The code stored in the diagram implements the steps related to model training in the embodiments of this application.

[0151] In this embodiment of the application, the training device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0152] It should be understood that the training device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps related to the training of the neutralization model provided in the embodiments of this application can also be implemented by the hardware system in the training device 520 without the function of executing instructions, which is not limited here.

[0153] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0154] (1) Neural Network

[0155] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0156] ;

[0157] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0158] (2) Deep Neural Networks

[0159] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: ,in, It is the input vector. It is the output vector. It is an offset vector. It is the weight matrix (also called coefficients). ( ) is the activation function. Each layer is simply an activation function applied to the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients... and offset vector The number of these parameters is therefore quite large. These parameters are defined in DNNs as follows: [as coefficients] For example: Suppose in a three-layer DNN, the linear coefficient from the fourth neuron in the second layer to the second neuron in the third layer is defined as... The superscript 3 represents the coefficient. The index corresponds to the level number, specifically the output index 2 for the third level and the input index 4 for the second level. In summary: Level L... The coefficients from the k-th neuron in layer 1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have... Parameters. In deep neural networks, more hidden layers allow the network to better depict complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can accomplish more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers in the trained deep neural network (a weight matrix formed by vectors W from many layers).

[0160] (3) Reinforcement learning (RL), also known as reinforcement learning, evaluation learning or reinforcement learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how an agent learns strategies to maximize rewards or achieve specific goals in the process of interacting with the environment.

[0161] The common model for reinforcement learning is the standard Markov decision process (MDP). Depending on the conditions, reinforcement learning can be categorized into model-based reinforcement learning (RL) and model-free reinforcement learning (RL), as well as active reinforcement learning (RL) and passive reinforcement learning (RL). Variations of reinforcement learning include inverse reinforcement learning, hierarchical reinforcement learning, and reinforcement learning for partially observable systems. Algorithms used to solve reinforcement learning problems can be divided into policy search algorithms and value function algorithms. Deep learning models can be used in reinforcement learning, forming deep reinforcement learning.

[0162] (4) Loss function

[0163] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0164] (5) Backpropagation algorithm

[0165] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0166] (6) Nash equilibrium

[0167] Also known as non-cooperative game equilibrium, it is an important term in game theory. In a game, regardless of the other players' strategies, one player will always choose a certain strategy; this strategy is called the dominant strategy. If any player's chosen strategy is optimal given that the strategies of all other players are fixed, then this combination is defined as a Nash equilibrium.

[0168] A strategy combination is called a Nash equilibrium, where each player's equilibrium strategy is to maximize their expected payoff, and all other players follow the same strategy.

[0169] (7) Reinforcement learning model

[0170] Reinforcement learning (RL), also known as reward learning, evaluation learning, or reinforcement learning, is a paradigm and methodology of machine learning used to describe and solve the problem of how an agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment.

[0171] The common model for reinforcement learning is the standard Markov decision process (MDP). Depending on the conditions, reinforcement learning can be categorized into model-based reinforcement learning (RL) and model-free reinforcement learning (RL), as well as active reinforcement learning (RL) and passive reinforcement learning (RL). Variations of reinforcement learning include inverse reinforcement learning, hierarchical reinforcement learning, and reinforcement learning for partially observable systems. Algorithms used to solve reinforcement learning problems can be divided into policy search algorithms and value function algorithms. Deep learning models can be used in reinforcement learning, forming deep reinforcement learning.

[0172] (8) Intelligent agent

[0173] An intelligent agent is a concept in the field of artificial intelligence. Any entity capable of independent thought and interaction with its environment can be abstracted as an intelligent agent. The basic characteristics of an intelligent agent are: it can react to changes in its environment and automatically adjust its behavior and state; different intelligent agents can also interact with other intelligent agents according to their own intentions.

[0174] In the application of reinforcement learning algorithms, online environments are often required for interaction to obtain data and for training. A common approach is to model real-world scenarios and generate virtual simulations of the online environment. However, even slight differences between the training environment and the real-world environment to be deployed can cause the trained algorithm to malfunction, resulting in suboptimal performance in real-world scenarios.

[0175] The aforementioned problems can be mitigated by improving the robustness of reinforcement learning algorithms. One approach is to introduce hypothetical interference into a virtual environment, training the reinforcement learning algorithm under such interference to enhance its ability to cope with interference, thereby improving its robustness and generalization. In other words, for the reinforcement learning model to be trained, an adversarial agent can be set up. The data output by this adversarial agent can be used to perform the task together with the output data of the reinforcement learning model, and the data output by the adversarial agent can serve as interference for performing the target task. Since the differences between the training and deployment environments are unpredictable, existing training methods mainly resist certain specific interferences. However, when changes in the real environment are inconsistent with hypothetical interferences, the algorithm's performance will deteriorate.

[0176] To solve the above problems, refer to Figure 4 , Figure 4 This is a flowchart illustrating a model training method provided in an embodiment of this application, as shown below. Figure 4 As shown in the embodiments of this application, a model training method includes:

[0177] 401. Processing first data through a first reinforcement learning model to obtain a first processing result; wherein the first data indicates the state of the target object, and the first processing result is used as control information when performing a target task on the target object.

[0178] In this step, the execution subject of step 401 can be a training device (for example, the training device can be a terminal device or a server), as described in the above embodiments, and will not be repeated here.

[0179] In one possible implementation, the training device can acquire the object of model training (the first reinforcement learning model) and training samples (the first data).

[0180] In one possible implementation, the first data is robot-related state information; the target task is robot posture control; and the first processing result is robot posture control information.

[0181] In one possible implementation, the robot's state information may include, but is not limited to, the robot's position, velocity, and information related to its environment (such as obstacle information). The robot's position and velocity may include information such as the state of each joint (position, angle, velocity, acceleration, etc.).

[0182] In one possible implementation, the first reinforcement learning model can obtain the robot's posture control information based on the input data. The posture control information can include the control information of each joint of the robot, and the robot's posture manipulation task can be performed based on the posture control information.

[0183] In one possible implementation, the first data is vehicle-related state information; the target task is autonomous driving of the vehicle; and the first processing result is vehicle driving control information.

[0184] In one possible implementation, vehicle-related state information may include, but is not limited to, the vehicle's position, speed, and information related to its environment (such as information about the road surface, obstacles, pedestrians, and surrounding vehicles).

[0185] In one possible implementation, the first reinforcement learning model can obtain driving control information for the vehicle based on the input data. The driving control information may include information such as the vehicle's speed, direction, and driving trajectory.

[0186] In one possible implementation, the first reinforcement learning model can be the initial model or the output of an iteration during model training.

[0187] In one possible implementation, during the feedforward process of model training, a first reinforcement learning model can process the first data to obtain a first processing result. This first processing result is used as control information when performing a target task on the target object. For example, if the target task is robot posture control, the first processing result is the robot's posture control information; or, if the target task is autonomous driving of a vehicle, the first processing result is the vehicle's driving control information.

[0188] Optionally, in one possible implementation, the first processing result can serve as a hard constraint imposed on the target object when performing the target task.

[0189] It should be understood that the reinforcement learning models in the embodiments of this application include, but are not limited to, deep neural networks, Bayesian neural networks, etc.

[0190] 402. The first data is processed by a first target neural network to obtain a second processing result; wherein the first processing result is used to perform a target task, and the second processing result is used as interference when performing the target task, the first target neural network is selected from a plurality of first neural networks, and each first neural network is an iterative result obtained by iteratively training a first initial neural network.

[0191] In one possible implementation, the training device can acquire an adversarial agent targeting the reinforcement learning model, which can output interference information for the target task.

[0192] In existing implementations, an adversarial agent can be trained to output interference information, which only targets the target task. In this embodiment, on the one hand, multiple adversarial agents can be trained to output interference information, and the interference information output by different adversarial agents can target different types of interference for the target task. On the other hand, when training the adversarial agents, not only can the adversarial agents obtained in the latest iteration be used to output interference for the target task, but also the historical training results of the adversarial agents (adversarial agents obtained in the historical iteration process) can be used to output interference for the target task. This allows for more effective interference for the target task that is adapted to different scenarios, thereby improving the training effect and generalization of the model.

[0193] It should be understood that the first target neural network in the embodiments of this application includes, but is not limited to, deep neural networks, Bayesian neural networks, etc.

[0194] In one possible implementation, when determining the adversarial agent for outputting interference information as a first reinforcement learning model, the first target neural network can be selected from the plurality of first neural networks, wherein each of the first neural networks is an iterative result obtained from the process of iteratively training a first initial neural network.

[0195] For example, during the iterative training of the first initial neural network, neural network 1, neural network 2, neural network 3, neural network 4, neural network 5, neural network 6, neural network 7, neural network 8, and neural network 9 can be obtained. When determining the adversarial agent used to output the interference information as the first reinforcement learning model, a neural network can be selected from the set [neural network 1, neural network 2, neural network 3, neural network 4, neural network 5, neural network 6, neural network 7, neural network 8, and neural network 9].

[0196] In one possible implementation, selecting the first target neural network from a plurality of first neural networks includes: selecting the first target neural network from the plurality of first neural networks based on a first selection probability corresponding to each of the plurality of first neural networks. That is, each first neural network can be configured with a probability (i.e., the first selection probability described above), and when selecting the first target neural network from the plurality of first neural networks, sampling can be performed based on the probability distribution corresponding to the plurality of first neural networks, and network selection can be performed based on the sampling results.

[0197] Next, we will introduce the probability of the first choice:

[0198] In one possible implementation, the processing result obtained by each first neural network is used as interference when performing the target task, and the first selection probability is positively correlated with the degree of interference of the corresponding first neural network output processing result on the target task. During the model update of the reinforcement learning model and the adversarial agent, a reward value can be obtained. This reward value can characterize both the quality of the data output by the reinforcement learning model in performing the target task and the degree of interference of the adversarial agent output interference information on the target task. The probability distribution corresponding to the first neural network can be updated based on the reward value, so that the first selection probability is positively correlated with the degree of interference of the corresponding first neural network output processing result on the target task. Through this method, on the one hand, for adversarial agents with a large output interference range, their corresponding sampling probability is larger, making them easier to sample and increasing the degree of interference to the reinforcement learning model. On the other hand, for adversarial agents with a small output interference range, although their corresponding sampling probability is smaller, they may still be sampled, which can increase the richness of the interference to the reinforcement learning model and improve the network's generalization ability.

[0199] In one possible implementation, the probability distribution described above can be a Nash equilibrium distribution. The probability distribution can be calculated using Nash equilibrium based on the reward value obtained by performing the target task using data and interference information received during the reinforcement learning model's feedforward. The probability distribution can be updated during iterations.

[0200] This application's embodiments control the behavior space of adversarial agents, altering the intensity of their interference, thus making the reinforcement learning strategy robust to both strong and weak interference. Furthermore, by introducing a game theory optimization framework and using historical strategies to increase the diversity of adversarial agents, the reinforcement learning strategy becomes even more robust to interference from different strategies.

[0201] In one possible implementation, during the feedforward process of model training, the first data can be processed by a first target neural network to obtain a second processing result, which is used as interference information when performing the target task.

[0202] For example, in a robot control scenario, the second processing result can be a force or torque applied to at least one joint on the robot. In another example, in an autonomous driving scenario, the second processing result can be information about obstacles or other obstacles that can affect the driving strategy applied to the road conditions of the vehicle.

[0203] In one possible implementation, to enhance the richness of the perturbations on the reinforcement learning model, multiple adversarial agents can be trained, and for each of the multiple adversarial agents, an adversarial agent that applies perturbations to the reinforcement learning model can be selected from multiple iterations during training.

[0204] For example, regarding the first initial neural network, during the iterative training process, neural networks A1, A2, A3, A4, A5, A6, A7, A8, and A9 can be obtained. When determining the adversarial agent used to output interference information as the first reinforcement learning model, a neural network can be selected from the set [neural networks A1, A2, A3, A4, A5, A6, A7, A8, and A9], which is the first target neural network in the above embodiment. Regarding the second initial neural network, which is different from the first initial neural network, during the iterative training process, neural networks B1, B2, B3, B4, B5, B6, B7, B8, and B9 can be obtained. When determining the adversarial agent used to output interference information as the first reinforcement learning model, a neural network can be selected from the set [neural networks B1, B2, B3, B4, B5, B6, B7, B8, and B9], which is the second target neural network. The data output by the first target neural network and the second target neural network can be used as interference information applied to the first reinforcement learning model.

[0205] Specifically, in one possible implementation, during the feedforward process based on the second target neural network, the first data can be processed by the second target neural network to obtain a fourth processing result; wherein the second processing result is used as interference when performing the target task, the second target neural network is selected from a plurality of second neural networks, each of the second neural networks being an iterative result obtained from the iterative training process of the second initial neural network; the first initial neural network and the second initial neural network are different.

[0206] In one possible implementation, the interference types of the second processing result and the fourth processing result are different.

[0207] For example, the type of interference can be the category of interference applied when performing the target task, such as applying force, applying torque, adding obstacles, changing road conditions, changing weather, etc.

[0208] In one possible implementation, the interference objects of the second processing result and the fourth processing result are different.

[0209] For example, a robot may include multiple joints, and applying forces to different joints or different groups of joints can be considered as dealing with different objects of interference. That is, the second processing result and the fourth processing result are forces applied to different joints or different groups of joints.

[0210] In one possible implementation, the first target neural network is used to determine the second processing result from a first numerical range based on the first data, and the second target neural network is used to determine the fourth processing result from a second numerical range based on the first data, wherein the second numerical range is different from the first numerical range.

[0211] For example, both the second and fourth processing results are forces applied to the robot joints. The maximum value of the force determined by the first target neural network is A1, and the maximum value of the force determined by the second target neural network is A2. A1 and A2 are different.

[0212] 403. Based on the first processing result and the second processing result, execute the target task to obtain the third processing result.

[0213] The first processing result can serve as a hard constraint when performing the target task, that is, the first processing result can serve as the control information that the target object needs to satisfy when performing the target task. The second processing result can serve as the interference applied to the target object when performing the target task. The third processing result can serve as the state of the target object when (or after) performing the target task. The third processing result can be used to determine the reward value.

[0214] It should be understood that the first processing result and the second processing result can be part of the data used to determine the third processing result. When other interference information besides the second processing result may also be included (such as the fourth processing result described in the above embodiments), the target task can be performed based on the first processing result, the second processing result and other processing results to obtain the third processing result.

[0215] 404. Based on the third processing result, update the first reinforcement learning model to obtain the updated first reinforcement learning model.

[0216] In one possible implementation, when updating the model, the first reinforcement learning model can be updated based on the third processing result to obtain the updated first reinforcement learning model.

[0217] For example, when updating the first reinforcement learning model, the cumulative reward obtained can be maximized. The update method can adopt a reinforcement learning algorithm with continuous action space, or alternatively, a trust region policy optimization (TRPO) algorithm can be adopted.

[0218] The first target neural network and the second target neural network can perform different adversarial tasks. In one possible implementation, the adversarial task required for this training can be selected from multiple adversarial tasks (exemplarily, in order).

[0219] In one possible implementation, historical policies of adversarial agents can be sampled from their historical policy set according to a Nash equilibrium distribution to be used against the reinforcement learning policy. In the training environment, the selected adversarial agent policy and the current reinforcement learning policy are deployed, and sampling is performed to obtain the required training samples. The obtained training samples are used to train the reinforcement learning policy. That is, the first reinforcement learning model can be updated based on the first processing result, the second processing result, and the fourth processing result to obtain an updated first reinforcement learning model (i.e., based on the first processing result, the second processing result, and the fourth processing result, the target task is executed to obtain a third processing result, and the first reinforcement learning model is updated based on the third processing result).

[0220] In one possible implementation, after sampling an adversarial agent for each adversarial task, the reinforcement learning policy and the updated policy of the adversarial agent can be added to the Nash equilibrium matrix, and the Nash equilibrium can be calculated to obtain the Nash equilibrium distribution of the reinforcement learning and adversarial agents (that is, the first selection probability mentioned above and the second selection probability introduced later). Specifically, updating the first reinforcement learning model based on the first processing result and the second processing result includes: obtaining the reward value corresponding to the target task based on the first processing result and the second processing result; updating the first reinforcement learning model based on the reward value; and then updating the first selection probability corresponding to the first target neural network based on the reward value.

[0221] In one possible implementation, during the iterative training of the adversarial agent, the reinforcement learning model participating in the current round can be selected from the historical iteration results of the reinforcement learning model. For example, this can be based on probability sampling, similar to the sampling process for the adversarial agent described in the above embodiments.

[0222] In one possible implementation, a second reinforcement learning model can be used to process the second data to obtain a fifth processing result; wherein the second reinforcement learning model is selected from a plurality of reinforcement learning models, including the updated first reinforcement learning model, and each reinforcement learning model is an iterative result obtained by iteratively training an initial reinforcement learning model; the second data indicates the state of the target object, and the fifth processing result is used as control information when performing the target task on the target object; the second data is processed by a third target neural network to obtain a sixth processing result; the third target neural network belongs to the plurality of first neural networks; the sixth processing result is used as interference information when performing the target task; the target task is performed according to the fifth processing result and the sixth processing result to obtain a seventh processing result; the third target neural network is updated according to the seventh processing result to obtain an updated third target neural network.

[0223] In one possible implementation, the second reinforcement learning model can be selected from the plurality of reinforcement learning models.

[0224] In one possible implementation, selecting the second reinforcement learning model from the plurality of reinforcement learning models includes: selecting the second reinforcement learning model from the plurality of reinforcement learning models based on a second selection probability corresponding to each reinforcement learning model among the plurality of reinforcement learning models.

[0225] In one possible implementation, the second selection probability is positively correlated with the positive performance of the corresponding reinforcement learning model's output when executing the target task. Specifically, during the model updates of the reinforcement learning model and the adversarial agent, a reward value is obtained. This reward value characterizes the quality of the reinforcement learning model's output data when executing the target task. The probability distribution corresponding to the reinforcement learning model can be updated based on the reward value, ensuring that the second selection probability is positively correlated with the positive performance of the corresponding reinforcement learning model's output when executing the target task.

[0226] In one possible implementation, historical policies of the reinforcement learning agent can be sampled from its historical policy set according to a Nash equilibrium distribution for use in updating the adversarial agent's policy. In the training environment, the selected reinforcement learning policy and the current adversarial agent policy are deployed, and sampling is performed to obtain the required training samples. These training samples are then used to train the adversarial agent's policy.

[0227] This application provides a model training method, comprising: processing first data through a first reinforcement learning model to obtain a first processing result; wherein the first data indicates the state of a target object, and the first processing result is used as control information when performing a target task on the target object; processing the first data through a first target neural network to obtain a second processing result; wherein the second processing result is used as interference information when performing the target task, the first target neural network is selected from a plurality of first neural networks, each of the first neural networks being an iterative result obtained by iteratively training a first initial neural network; performing the target task according to the first processing result and the second processing result to obtain a third processing result; and updating the first reinforcement learning model according to the third processing result to obtain an updated first reinforcement learning model. Through the above method, when training an adversarial agent, not only can the adversarial agent obtained in the latest iteration be used to output interference for the target task, but also the historical training results of the adversarial agent (adversarial agents obtained in the historical iteration process) can be used to output interference for the target task, thereby obtaining more effective interference adapted to different scenarios for the target task, thereby improving the training effect and generalization of the model.

[0228] The following section uses a robot as the target object and robot control as the target task as an example to introduce a software architecture of an embodiment of this application:

[0229] Reference Figure 5 , Figure 5 A robot control system is shown, such as Figure 5As shown, the robot control system may include: a state perception and processing module, a robust decision-making module, and a robot control module.

[0230] Regarding the state perception and processing module: This module's function is to perceive information about the robot (such as information describing the state of the target object, as described in the above embodiments, such as first data, second data, etc.). Specifically, it integrates information transmitted from various sensors to determine the robot's own state, including basic robot information (position, velocity), the state of each joint (position, angle, velocity, acceleration), and other information, and then transmits this information to the decision-making module.

[0231] Regarding the robustness decision-making module: This module's function is to output upper-level behavioral decisions for a future period based on the robot's current state and the task being performed (e.g., the control information described in the above embodiment when performing a target task on the target object). Specifically, this module, based on the robot's current state output by the state perception and processing module, through... Figure 4 The corresponding method can output behavioral decisions for a future period of time and pass them to the robot control module.

[0232] Regarding the robot control module: This module controls the robot's joints and executes the behaviors output by the robust decision-making module to control the robot's movement.

[0233] Specifically, refer to Figure 6 , Figure 6 This document illustrates the process of applying the model training method described in this application to a robot control simulation scenario. The robot employs the model training method described in this application, utilizing a multi-task framework and Boolean-theoretic optimization, to ultimately output behavioral decisions that maximize the robot's forward speed and obtain greater rewards. The implementation method is described in detail below.

[0234] S1. Input parameters for multi-task learning Initialize reinforcement learning strategy Initialize the adversarial agent policy for each task i. ,choose The i-th parameter in the equation serves as the strategy of the adversarial agent. The action space parameters are used to construct multiple tasks, in which an adversarial agent can apply a perturbation force to the simulated robot's body. The initial Nash equilibrium distribution can be a uniform distribution.

[0235] S2. According to Select the corresponding adversarial agents in turn as the current adversarial task.

[0236] S3. Based on the distribution of adversarial agents in the Nash equilibrium, sample the historical strategy of one adversarial agent. This serves as the adversarial strategy for the current adversarial task, and the adversarial strategy is deployed to the training environment.

[0237] S4. Based on reinforcement learning strategies and adversarial agent strategies In the training environment, the robot is controlled to collect M samples. ,in These represent the behavioral output of the reinforcement learning strategy and the behavior of the adversarial agent, respectively.

[0238] S5. On reinforcement learning strategies The update is performed, and the objective function for the update is:

[0239] ;

[0240] That is, to maximize the cumulative reward obtained, the update method can adopt a reinforcement learning algorithm in the continuous action space, or optionally, a trust region policy optimization (TRPO) algorithm.

[0241] S6. Based on the historical distribution of reinforcement learning policies in Nash equilibrium, sample the historical policy of a reinforcement learning agent. As a reinforcement learning strategy that needs to be interfered with in current adversarial agent strategies, this reinforcement learning strategy is deployed to the training environment.

[0242] S7. Based on reinforcement learning strategies and adversarial agent strategies In the training environment, the robot is controlled to collect M samples. .

[0243] S8. Adversarial Agent Strategy The update is performed, and the objective function for the update is:

[0244] ;

[0245] That is, minimizing the cumulative reward obtained by the reinforcement learning policy, which hinders the reinforcement learning agent from achieving the goal. The update method can adopt a reinforcement learning algorithm with continuous action space, or optionally, the Trusted Region Policy Optimization (TRPO) algorithm.

[0246] S9. Every k steps, for each task, add the reinforcement learning policy and the updated policy of the adversarial agent to the Nash equilibrium matrix. By iterating through and testing the performance of the newly added policy and the existing historical policies in the training environment, obtain the Nash equilibrium value matrix of the newly added policy, and calculate the Nash equilibrium based on the value matrix to obtain the Nash equilibrium distribution of the reinforcement learning and adversarial agents.

[0247] Determine whether the current task has ended. If it has not ended, proceed to step S2; otherwise, proceed to step S10.

[0248] S10. Deploy the trained reinforcement learning policy to a test environment that differs from the training environment to test its robustness.

[0249] The embodiments described above employ a robust reinforcement learning control framework based on multi-task learning and game theory. By modifying the action space of the adversarial agent to construct multiple adversarial tasks, the robustness of the reinforcement learning algorithm is improved. Furthermore, a game theory-based optimization framework is introduced to select the most suitable adversarial strategy based on historical policy performance during the training process of each task, making the reinforcement learning strategy even more robust.

[0250] It should be understood that the game theory optimization framework in the embodiments of this application includes, but is not limited to, policy-space response oracles (PSRO); the training of reinforcement learning models includes, but is not limited to, sampling reinforcement learning algorithms, such as Trusted Space Policy Optimization (TRPO) and Proximal Policy Optimization (PPO).

[0251] This application provides a model training method, the method comprising: processing first data through a first reinforcement learning model to obtain a first processing result; wherein the first data indicates the state of a target object, and the first processing result is used as control information when performing a target task on the target object; processing the first data through a first target neural network to obtain a second processing result; wherein the second processing result is used as interference information when performing the target task, the first target neural network is selected from a plurality of first neural networks, each of the first neural networks being an iterative result obtained by iteratively training a first initial neural network; performing the target task according to the first processing result and the second processing result to obtain a third processing result; and updating the first reinforcement learning model according to the third processing result to obtain an updated first reinforcement learning model. In existing implementations, an adversarial agent can be trained to output interference information, which only targets the target task. In this embodiment, on the one hand, multiple adversarial agents can be trained to output interference information, and the interference information output by different adversarial agents can target different types of interference for the target task. On the other hand, when training the adversarial agents, not only can the adversarial agents obtained in the latest iteration be used to output interference for the target task, but also the historical training results of the adversarial agents (adversarial agents obtained in the historical iteration process) can be used to output interference for the target task. This allows for more effective interference for the target task that is adapted to different scenarios, thereby improving the training effect and generalization of the model.

[0252] Reference Figure 7 , Figure 7 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application, as shown below. Figure 7 As shown, the device 700 includes:

[0253] The data processing module 701 is used to process the first data through a first reinforcement learning model to obtain a first processing result; wherein the first data indicates the state of the target object, and the first processing result is used as control information when performing a target task on the target object;

[0254] The first data is processed by a first target neural network to obtain a second processing result; wherein the second processing result is used as interference information when performing the target task, the first target neural network is selected from a plurality of first neural networks, and each first neural network is an iterative result obtained by iteratively training a first initial neural network;

[0255] Based on the first processing result and the second processing result, the target task is executed to obtain a third processing result;

[0256] The specific description of the data processing module 701 can be found in the descriptions of steps 401, 402 and 403 in the above embodiments, and will not be repeated here.

[0257] The model update module 702 is used to update the first reinforcement learning model according to the third processing result, so as to obtain the updated first reinforcement learning model.

[0258] The specific description of the model update module 702 can be found in the description of step 404 in the above embodiment, and will not be repeated here.

[0259] In existing implementations, an adversarial agent can be trained to output interference information, which only targets the target task. In this embodiment, on the one hand, multiple adversarial agents can be trained to output interference information, and the interference information output by different adversarial agents can target different types of interference for the target task. On the other hand, when training the adversarial agents, not only can the adversarial agents obtained in the latest iteration be used to output interference for the target task, but also the historical training results of the adversarial agents (adversarial agents obtained in the historical iteration process) can be used to output interference for the target task. This allows for more effective interference for the target task that is adapted to different scenarios, thereby improving the training effect and generalization of the model.

[0260] In one possible implementation,

[0261] The target object is a robot; the target task is the robot's posture control; and the first processing result is the robot's posture control information; or...

[0262] The target object is a vehicle; the target task is the autonomous driving of the vehicle; and the first processing result is the vehicle's driving control information.

[0263] In one possible implementation, the first target neural network is selected from a plurality of first neural networks based on a first selection probability corresponding to each of the plurality of first neural networks.

[0264] In one possible implementation, the model update module is specifically used for:

[0265] Based on the third processing result, the reward value corresponding to the target task is obtained;

[0266] The first reinforcement learning model is updated based on the reward value;

[0267] The model update module is also used for:

[0268] The first selection probability corresponding to the first target neural network is updated based on the reward value.

[0269] In one possible implementation, the data processing module is further configured to:

[0270] The first data is processed by a second target neural network to obtain a fourth processing result; wherein the fourth processing result is used as interference information when performing the target task, the second target neural network is selected from a plurality of second neural networks, each of the second neural networks being an iterative result obtained by iteratively training a second initial neural network; the first initial neural network and the second initial neural network are different;

[0271] The data processing module is specifically used for:

[0272] Based on the first processing result, the fourth processing result, and the second processing result, the target task is executed to obtain the third processing result.

[0273] In one possible implementation, the interference types of the second processing result and the fourth processing result are different; or,

[0274] The interference targets of the second processing result and the fourth processing result are different; or,

[0275] The first target neural network is used to determine the second processing result from a first numerical range based on the first data, and the second target neural network is used to determine the fourth processing result from a second numerical range based on the first data, wherein the second numerical range is different from the first numerical range.

[0276] In one possible implementation, the data processing module is further configured to:

[0277] The second reinforcement learning model processes the second data to obtain a fifth processing result; wherein the second reinforcement learning model is selected from multiple reinforcement learning models, including the updated first reinforcement learning model, and each reinforcement learning model is an iterative result obtained by iteratively training an initial reinforcement learning model; the second data indicates the state of the target object, and the fifth processing result is used as control information when performing the target task on the target object;

[0278] The second data is processed by a third target neural network to obtain a sixth processing result; the third target neural network belongs to the plurality of first neural networks; the sixth processing result is used as interference information when performing the target task.

[0279] Based on the fifth and sixth processing results, the target task is executed to obtain the seventh processing result;

[0280] The model update module is also used for:

[0281] Based on the seventh processing result, the third target neural network is updated to obtain the updated third target neural network.

[0282] In one possible implementation, the second reinforcement learning model is selected from multiple reinforcement learning models based on the second selection probability corresponding to each reinforcement learning model.

[0283] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 8 , Figure 8 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 800 can specifically be a mobile phone, tablet, laptop, smart wearable device, etc., and is not limited thereto. Specifically, the execution device 800 includes: a receiver 801, a transmitter 802, a processor 803, and a memory 804 (wherein the execution device 800 may have one or more processors 803). Figure 8 (Taking a processor as an example), the processor 803 may include an application processor 8031 ​​and a communication processor 8032. In some embodiments of this application, the receiver 801, transmitter 802, processor 803, and memory 804 may be connected via a bus or other means.

[0284] Memory 804 may include read-only memory and random access memory, and provides instructions and data to processor 803. A portion of memory 804 may also include non-volatile random access memory (NVRAM). Memory 804 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0285] The processor 803 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0286] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 803. Processor 803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 803 or by instructions in software form. Processor 803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 803 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 804, and processor 803 reads the information from memory 804 and, in conjunction with its hardware, completes the steps of the above method.

[0287] Receiver 801 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 802 can be used to output digital or character information; transmitter 802 can also be used to send instructions to the disk group to modify the data in the disk group.

[0288] In one embodiment of this application, the processor 803 is configured to execute via Figure 4 The steps for obtaining the model using the model training method in the corresponding embodiment.

[0289] This application also provides a server; please refer to [link / reference]. Figure 9 , Figure 9This is a schematic diagram of a server structure provided in an embodiment of this application. Specifically, server 900 is implemented by one or more servers. Server 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 99 (e.g., one or more processors) and memory 932, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 942 or data 944. The memory 932 and storage media 930 can be temporary or persistent storage. The program stored in storage media 930 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 99 may be configured to communicate with storage media 930 and execute the series of instruction operations in storage media 930 on server 900.

[0290] Server 900 may also include one or more power supplies 991, one or more wired or wireless network interfaces 950, one or more input / output interfaces 958; or one or more operating systems 941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0291] In this embodiment, the central processing unit 99 is used to execute... Figure 4 The steps of the model training method in the corresponding embodiment.

[0292] This application also provides a computer program product including computer-readable instructions, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0293] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0294] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the model training method described in the above embodiments, or to cause the chip within the training device to execute the steps related to model training in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0295] For details, please refer to Figure 10 , Figure 10 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1000. The NPU 1000 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1003, which is controlled by the controller 1004 to extract matrix data from the memory and perform multiplication operations.

[0296] In some implementations, the arithmetic circuit 1003 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1003 is a two-dimensional pulsating array. The arithmetic circuit 1003 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1003 is a general-purpose matrix processor.

[0297] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1008.

[0298] Unified memory 1006 is used to store input and output data. Weight data is directly transferred to weight memory 1002 via Direct Memory Access Controller (DMAC) 1005. Input data is also transferred to unified memory 1006 via DMAC.

[0299] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1009.

[0300] The Bus Interface Unit (BIU) 1010 is used by the instruction fetch memory 1009 to fetch instructions from external memory, and also by the memory access controller 1005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0301] The DMAC is mainly used to move input data from external memory DDR to unified memory 1006, or to weight data to weight memory 1002, or to input data to input memory 1001.

[0302] The vector computation unit 1007 includes multiple processing units that further process the output of the computation circuit when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as Batch Normalization, pixel-level summation, and upsampling of feature planes.

[0303] In some implementations, the vector computation unit 1007 can store the processed output vector in the unified memory 1006. For example, the vector computation unit 1007 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1003, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1003, for example, for use in subsequent layers of the neural network.

[0304] The instruction fetch buffer 1009 connected to the controller 1004 is used to store the instructions used by the controller 1004;

[0305] Unified memory 1006, input memory 1001, weight memory 1002, and instruction fetch memory 1009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0306] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0307] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0308] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0309] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0310] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A model training method, characterized in that, The method includes: A first reinforcement learning model processes first data to obtain a first processing result; wherein the first data indicates the state of the target object, and the first processing result is used as control information when performing a target task on the target object; The first data is processed by a first target neural network to obtain a second processing result; wherein the second processing result is used as interference information when performing the target task, the first target neural network is selected from a plurality of first neural networks, and each first neural network is an iterative result obtained by iteratively training a first initial neural network; Based on the first processing result and the second processing result, the target task is executed to obtain a third processing result; Based on the third processing result, the first reinforcement learning model is updated to obtain an updated first reinforcement learning model; the target object is a robot; the target task is robot posture control, and the first processing result is the robot's posture control information; or... The target object is a vehicle; the target task is the autonomous driving of the vehicle; and the first processing result is the vehicle's driving control information.

2. The method according to claim 1, characterized in that, The first target neural network is selected from multiple first neural networks based on the first selection probability corresponding to each of the multiple first neural networks.

3. The method according to claim 2, characterized in that, The processing result obtained by each first neural network is used as interference when performing the target task, and the first selection probability is positively correlated with the degree of interference of the processing result output by the corresponding first neural network on the target task.

4. The method according to claim 2, characterized in that, The step of updating the first reinforcement learning model based on the third processing result includes: Based on the third processing result, the reward value corresponding to the target task is obtained; The first reinforcement learning model is updated based on the reward value; The method further includes: The first selection probability corresponding to the first target neural network is updated based on the reward value.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The first data is processed by a second target neural network to obtain a fourth processing result; wherein the fourth processing result is used as interference information when performing the target task, the second target neural network is selected from a plurality of second neural networks, each of the second neural networks being an iterative result obtained by iteratively training a second initial neural network; the first initial neural network and the second initial neural network are different; The step of executing the target task based on the first processing result and the second processing result to obtain a third processing result includes: Based on the first processing result, the fourth processing result, and the second processing result, the target task is executed to obtain the third processing result.

6. The method according to claim 5, characterized in that, The interference types of the second processing result and the fourth processing result are different; or, The interference targets of the second processing result and the fourth processing result are different; or, The first target neural network is used to determine the second processing result from a first numerical range based on the first data, and the second target neural network is used to determine the fourth processing result from a second numerical range based on the first data, wherein the second numerical range is different from the first numerical range.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The second reinforcement learning model processes the second data to obtain a fifth processing result; wherein the second reinforcement learning model is selected from multiple reinforcement learning models, including the updated first reinforcement learning model, and each reinforcement learning model is an iterative result obtained by iteratively training an initial reinforcement learning model; the second data indicates the state of the target object, and the fifth processing result is used as control information when performing the target task on the target object; The second data is processed by a third target neural network to obtain a sixth processing result; the third target neural network belongs to the plurality of first neural networks; the sixth processing result is used as interference information when performing the target task. Based on the fifth and sixth processing results, the target task is executed to obtain the seventh processing result; Based on the seventh processing result, the third target neural network is updated to obtain the updated third target neural network.

8. The method according to claim 7, characterized in that, The second reinforcement learning model is selected from multiple reinforcement learning models based on the second selection probability corresponding to each reinforcement learning model.

9. A model training device, characterized in that, The device includes: A data processing module is used to process first data through a first reinforcement learning model to obtain a first processing result; wherein the first data indicates the state of a target object, and the first processing result is used as control information when performing a target task on the target object; The first data is processed by a first target neural network to obtain a second processing result; wherein the second processing result is used as interference information when performing the target task, the first target neural network is selected from a plurality of first neural networks, and each first neural network is an iterative result obtained by iteratively training a first initial neural network; Based on the first processing result and the second processing result, the target task is executed to obtain a third processing result; A model update module is used to update the first reinforcement learning model based on the third processing result to obtain an updated first reinforcement learning model; the target object is a robot; the target task is robot posture control; and the first processing result is the robot's posture control information; or... The target object is a vehicle; the target task is the autonomous driving of the vehicle; and the first processing result is the vehicle's driving control information.

10. The apparatus according to claim 9, characterized in that, The first target neural network is selected from multiple first neural networks based on the first selection probability corresponding to each of the multiple first neural networks.

11. The apparatus according to claim 10, characterized in that, The processing result obtained by each first neural network is used as interference when performing the target task, and the first selection probability is positively correlated with the degree of interference of the processing result output by the corresponding first neural network on the target task.

12. The apparatus according to claim 10, characterized in that, The model update module is specifically used for: Based on the third processing result, the reward value corresponding to the target task is obtained; The first reinforcement learning model is updated based on the reward value; The model update module is also used for: The first selection probability corresponding to the first target neural network is updated based on the reward value.

13. The apparatus according to any one of claims 9 to 12, characterized in that, The data processing module is also used for: The first data is processed by a second target neural network to obtain a fourth processing result; wherein the fourth processing result is used as interference information when performing the target task, the second target neural network is selected from a plurality of second neural networks, each of the second neural networks being an iterative result obtained by iteratively training a second initial neural network; the first initial neural network and the second initial neural network are different; The data processing module is specifically used for: Based on the first processing result, the fourth processing result, and the second processing result, the target task is executed to obtain the third processing result.

14. The apparatus according to claim 13, characterized in that, The interference types of the second processing result and the fourth processing result are different; or, The interference targets of the second processing result and the fourth processing result are different; or, The first target neural network is used to determine the second processing result from a first numerical range based on the first data, and the second target neural network is used to determine the fourth processing result from a second numerical range based on the first data, wherein the second numerical range is different from the first numerical range.

15. The apparatus according to any one of claims 9 to 12, characterized in that, The data processing module is also used for: The second reinforcement learning model processes the second data to obtain a fifth processing result; wherein the second reinforcement learning model is selected from multiple reinforcement learning models, including the updated first reinforcement learning model, and each reinforcement learning model is an iterative result obtained by iteratively training an initial reinforcement learning model; the second data indicates the state of the target object, and the fifth processing result is used as control information when performing the target task on the target object; The second data is processed by a third target neural network to obtain a sixth processing result; the third target neural network belongs to the plurality of first neural networks; the sixth processing result is used as interference information when performing the target task. Based on the fifth and sixth processing results, the target task is executed to obtain the seventh processing result; The model update module is also used for: Based on the seventh processing result, the third target neural network is updated to obtain the updated third target neural network.

16. The apparatus according to claim 15, characterized in that, The second reinforcement learning model is selected from multiple reinforcement learning models based on the second selection probability corresponding to each reinforcement learning model.

17. A model training device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to retrieve the code and perform the method as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 8.

Images