A data processing method and related equipment

Abstract

A data processing method relates to the field of artificial intelligence, the method comprising: processing a first state through a decision model to obtain first control information; processing the first state and the first control information through multiple machine learning models to predict multiple second states of the decision model; predicting the first long-term benefits corresponding to each second state based on the multiple second states; the multiple first long-term benefits are used to select a target model from the multiple machine learning models; and the second state obtained by the target model is used to update the decision model. The present application selects a machine learning model for use as a prediction state based on the long-term benefits of the second state obtained by the machine learning model, which can characterize the uncertainty of the prediction state in real time, and adaptively regulates the selection of the machine learning model, which can more accurately select a prediction state with a certain degree of conservatism, thereby improving the performance of the decision model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a data processing method and related equipment. Background Art

[0002] In the typical paradigm of policy optimization, an agent (for example, the decision model in this application can be a reinforcement learning model that can predict the control strategy based on the state of the target object) can actively interact with the environment and receive feedback data from the environment, and use the feedback to promote the agent to continuously improve the strategy. However, in practice, the cost required for the agent to interact with the environment to obtain data is often very expensive or even unsafe (for example, in business and medicine). Therefore, the adverse effects of the data collection process have become a major obstacle to the deployment of policy optimization in practical application problems.

[0003] Because policy optimization is often infeasible in certain applications, offline policy optimization (PO) is often employed. Offline PO involves learning the optimal policy using a fixed dataset derived from the interaction of an existing (unknown) policy with the environment, without allowing the agent to interact with the environment. Since the agent and the environment do not interact, the learning method is called "offline." The policy corresponding to the fixed dataset is called an action policy.

[0004] In existing implementations, during the offline strategy optimization of the decision model, due to the lack of training samples (the state of the target object, etc.), in each step, based on the control strategy obtained by the decision model and the current state of the target object, multiple pre-trained machine learning models are used to predict the state of the target object. However, since the state of the target object is predicted, it is often not accurate, so a conservative value function estimation method is used to fit the lower bound. Specifically, the uncertainty distribution of the prediction results of multiple machine learning models can be calculated, and the size of the uncertainty distribution can be controlled by a preset coefficient to achieve the fitting of the lower bound. However, in order to ensure that the predicted state does not have serious errors, the setting of the coefficients is often too conservative, and the inconsistency between the dimension and state of the uncertainty distribution will also lead to incorrect numerical results, resulting in poor performance of the trained decision model. Summary of the Invention

[0005] This application provides a data processing method that can improve the performance of decision models.

[0006] In the first aspect, the present application provides a data processing method, which includes: processing a first state through a decision model to obtain first control information; wherein the first state indicates the state of a target object, and the first control information is used as control information when performing a target task on the target object; processing the first state and the first control information through multiple machine learning models to predict multiple second states of the decision model; each second state is a predicted state after controlling the target object in the first state through the first control information; based on the multiple second states, predicting the first long-term benefit corresponding to each second state; the multiple first long-term benefits are used to select a target model from multiple machine learning models; the second state obtained by the target model is used to update the decision model.

[0007] In existing technologies, the uncertainty distribution of machine learning model outputs and fixed coefficients are used to control the size of the distribution. To ensure that the predicted state does not make serious errors, the coefficient settings are often overly conservative, resulting in poor performance of the trained model. Furthermore, the impact of machine learning model uncertainty on long-term returns is not precisely described. This inaccuracy is reflected in the problem of incorrect numerical results due to inconsistent dimensions and overly conservative approaches in low model confidence levels. Furthermore, the existing technology generally uses fixed coefficients to control the size of uncertainty, lacking a global perspective on the entire sequential decision-making process. This lack of global perspective has a negative impact on policy optimization, and it cannot dynamically update the posterior probability based on the training results.

[0008] In an embodiment of the present application, the machine learning model used as the prediction state is selected based on the long-term benefits of the second state obtained by the machine learning model. This can characterize the uncertainty of the prediction state in real time, and adaptively adjust the selection of the machine learning model. It can more accurately select a prediction state with a certain degree of conservatism, thereby improving the performance of the decision-making model.

[0009] In one possible implementation, the decision model can be a reinforcement learning model, an artificial neural network, a deep neural network, a Bayesian neural network, etc.

[0010] In one possible implementation, the target object is a robot; the target task is posture manipulation of the robot, and the first control information is posture control information of the robot; or, the target object is a vehicle; the target task is automatic driving of the vehicle, and the first control information is driving control information of the vehicle.

[0011] In one possible implementation, a parameter N can be preset, with the target model being the Nth model with the highest long-term returns among multiple machine learning models, sorted by size. A larger N, meaning a model with smaller long-term returns, results in more conservative predictions, potentially decreasing accuracy but increasing the likelihood of accurate predictions.

[0012] In one possible implementation, in order to make the prediction results more conservative, a machine learning model with a low long-term return ranking can be selected from multiple machine learning models. For example, a model with the Nth-ranked long-term return among multiple machine learning models can be selected, specifically the model with the smallest long-term return among multiple machine learning models, the model with the second-to-last smallest long-term return, or the model with the third-to-last smallest long-term return, etc.

[0013] In one possible implementation, a model set including multiple machine learning models can be trained. Multiple machine learning models constitute the model set. A sampling probability can be set for each machine learning model in the model set. In each training step, the machine learning models in the model set can be probabilistically sampled according to the sampling probability to obtain multiple machine learning models introduced in step 402.

[0014] In one possible implementation, each sampling probability is related to the correlation between the processing capabilities of the corresponding machine learning model and the decision model. For example, the probability set by the machine learning model can be positively correlated with the processing capabilities of the decision model. This makes machine learning models with a higher correlation with the processing capabilities of the decision model more likely to be sampled, making the subsequent predicted state less conservative and achieving a higher probability of obtaining a more accurate prediction result, but at the expense of a lower degree of conservatism in the prediction result.

[0015] In a possible implementation, each sampling probability is a preset probability value, which may be given by an expert or an experience value, for example.

[0016] In one possible implementation, different machine learning models in the plurality of machine learning models have different network structures or different parameter initialization values. Due to the differences in network structures or initialization methods, different machine learning models can predict different results for the same input.

[0017] In one possible implementation, the method also includes: predicting the second long-term benefit corresponding to the first state through a benefit evaluation function based on the first state; predicting the long-term benefit corresponding to each second state based on multiple second states, including: predicting the first long-term benefit corresponding to each second state through a benefit evaluation function based on multiple second states; updating the benefit evaluation function based on the difference between the first long-term benefit and the second long-term benefit.

[0018] In a possible implementation, the benefit evaluation function may be a function such as a Q function for performing long-term benefit evaluation.

[0019] In a second aspect, the present application provides a data processing device, comprising:

[0020] a processing module, configured to process the first state through a decision model to obtain first control information; wherein the first state indicates a state of the target object, and the first control information is used as control information when performing a target task on the target object;

[0021] Processing the first state and the first control information through multiple machine learning models to predict multiple second states of the decision model; each second state is a predicted state after controlling the target object in the first state through the first control information;

[0022] The profit prediction module is used to predict the first long-term profit corresponding to each second state based on multiple second states; the multiple first long-term profits are used to select a target model from multiple machine learning models; the second state obtained by the target model is used to update the decision model.

[0023] In one possible implementation,

[0024] The target object is a robot; the target task is posture control of the robot, and the first control information is posture control information of the robot; or,

[0025] The target object is a vehicle; the target task is automatic driving of the vehicle, and the first control information is driving control information of the vehicle.

[0026] In one possible implementation, the target model is the Nth long-term return model among multiple machine learning models sorted by size, where N is a preset parameter.

[0027] In one possible implementation, the model with the Nth long-term return is specifically the model with the smallest long-term return, the model with the second smallest long-term return, or the model with the third smallest long-term return among multiple machine learning models.

[0028] In one possible implementation, multiple machine learning models are sampled from a model set by probabilistic sampling, where each machine learning model in the model set corresponds to a sampling probability.

[0029] In one possible implementation, each sampling probability is related to the correlation between the processing capabilities of the corresponding machine learning model and decision model.

[0030] In a possible implementation, each sampling probability is a preset probability value.

[0031] In one possible implementation, different machine learning models among the multiple machine learning models have different network structures or different parameter initialization values.

[0032] In a possible implementation, the revenue prediction module is further configured to:

[0033] According to the first state, the second long-term benefit corresponding to the first state is predicted by using the benefit evaluation function;

[0034] Based on multiple second states, predict the long-term benefits corresponding to each second state, including:

[0035] The revenue forecast module is specifically used to:

[0036] The profit evaluation function is updated according to the difference between the first long-term profit and the second long-term profit.

[0037] In a third aspect, an embodiment of the present application provides a data processing device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to perform the first aspect and any optional method thereof.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method thereof.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.

[0040] In a sixth aspect, the present application provides a chip system, which includes a processor for supporting a model training device to implement some or all of the functions involved in the above aspects, such as sending or processing data involved in the above methods; or information. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for executing the device or training the device. The chip system can be composed of a chip or can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of an application architecture;

[0042] Figure 2 This is a schematic diagram of an application architecture;

[0043] Figure 3 This is a schematic diagram of an application architecture;

[0044] Figure 4 An example of a data processing method provided in an embodiment of the present application is shown;

[0045] Figure 5 An example of a data processing method provided in an embodiment of the present application is shown;

[0046] Figure 6 An example of a data processing device provided in an embodiment of the present application is shown;

[0047] Figure 7 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0048] Figure 8 This is a structural diagram of a server provided in an embodiment of the present application;

[0049] Figure 9 A schematic diagram of the structure of the chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0051] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0053] As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation, not as terms of degree, and are intended to take into account the inherent variations in measurements or calculations that one of ordinary skill in the art would recognize. Furthermore, the use of "may" when describing embodiments of the present invention refers to "one or more possible embodiments." As used herein, the terms "use," "using," and "used" may be considered synonymous with the terms "utilize," "utilizing," and "utilized," respectively. Additionally, the term "exemplary" is intended to refer to an example or illustration.

[0054] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The following diagram illustrates a structural diagram of the AI framework. This framework is explained below from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it encompasses the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed progression from "data-information-knowledge-wisdom." The "IT value chain," encompassing the entire process from the underlying infrastructure of human intelligence, information (provided and processed by technology), to the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0055] (1) Infrastructure

[0056] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0057] (2) Data

[0058] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0059] (3) Data processing

[0060] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0061] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0062] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0063] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0064] (4) General ability

[0065] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0066] (5) Smart products and industry applications

[0067] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0068] With the development of artificial intelligence, many tasks that require human intervention are gradually being replaced by smart terminals. Therefore, the smart terminals need to be equipped with the skills needed to complete the tasks, as well as task-specific neural networks, so as to enable the completion of specific tasks through the smart terminals. Specifically, this can be applied to mobile smart terminals. For example, in the field of autonomous driving, driving operations that were originally performed by humans can be replaced by smart cars. In this case, smart cars need to be equipped with a large number of driving skills and neural networks for driving skills. As another example, in the field of freight, handling operations that were originally performed by humans can be replaced by handling robots. In this case, handling robots need to be equipped with a large number of handling skills and neural networks for handling skills. It can also be applied to smart terminals that do not have mobile operations. As an example, on an assembly line for parts processing, the parts grabbing operations that were originally performed by humans can be completed by smart robotic arms. In this case, the smart robotic arms need to be equipped with grabbing skills and neural networks for the grabbing skills. Different grabbing skills may have different grabbing angles, displacements of smart robotic arms, etc. As another example, in the field of automatic cooking, the cooking operations that were originally performed by humans can be completed by smart robotic arms. In this case, the smart robotic arms need to be equipped with cooking skills such as raw material grabbing skills and stir-frying skills, as well as neural networks for cooking skills, etc. Other application scenarios are not enumerated here.

[0069] In order to better understand the solution of the embodiment of the present application, Figure 2 and Figure 3 A brief introduction to the possible implementation architecture of the embodiments of the present application is given.

[0070] Figure 2Schematic diagram of a computing system for performing model training in an embodiment of the present application. The computing system includes a terminal device 102 (exemplary, terminal device 102 may not be included) and a server 130 (also referred to as a central node) coupled via a network communication. The terminal device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smart phone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0071] The terminal device 102 may include a processor 112 and a memory 114. The processor 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a controller, a microcontroller, etc.). The memory 114 may include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM). The memory 114 may store data 116 and instructions 118 executed by the processor 112 to enable the terminal device 102 to perform operations.

[0072] In some embodiments, the memory 114 can store one or more models 120. For example, the model 120 can be or can additionally include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network can include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks.

[0073] In some implementations, one or more models 120 may be received from a server 130 over a network, stored in memory 114 , and then used or otherwise implemented by one or more processors 112 .

[0074] The terminal device 102 may also include one or more user input components 122 for receiving user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0075] The terminal device 102 may also include a communication interface 123, and the terminal device 102 may be connected to the server 130 through the communication interface 123. The server 130 may include a communication interface 133, and the terminal device 102 may be connected to the communication interface 133 of the server 130 through the communication interface 123, thereby realizing data interaction between the terminal device 102 and the server 130.

[0076] The server 130 may include a processor 132 and a memory 134. The processor 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a controller, a microcontroller, etc.). The memory 134 may include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM). The memory 134 may store data 136 and instructions 138 executed by the processor 132 to enable the server 130 to perform operations.

[0077] As described above, memory 134 may store one or more machine learning models 140. For example, model 140 may be or may further include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0078] It should be understood that the model training method in the embodiment of the present application involves AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and the server is not limited to Figure 2 The processor shown is combined with the memory architecture. Figure 3The system architecture provided in the embodiments of the present application is introduced in detail.

[0079] Figure 3 This is a schematic diagram of the system architecture provided in the embodiment of this application. Figure 3 As shown, the system architecture 500 includes an execution device 510 , a training device 520 , a database 530 , a client device 540 , a data storage system 550 , and a data collection system 560 .

[0080] The execution device 510 includes a calculation module 511, an I / O interface 512, a pre-processing module 513, and a post-processing module 514. The calculation module 511 may include the target model / rule 501, and the pre-processing module 513 and the post-processing module 514 are optional.

[0081] Data acquisition device 560 is used to collect training samples. The training samples can be, for example, a first state, where the first state can be state information related to a target object (e.g., a robot, a vehicle, etc.), state information related to the vehicle, etc. After collecting the training samples, data acquisition device 560 stores them in database 530.

[0082] The training device 520 can maintain the training samples in the database 530 to train the neural network to be trained (such as the reinforcement learning model in the embodiment of the present application) to obtain the target model / rule 501.

[0083] It should be noted that, in actual applications, the training samples maintained in the database 530 may not all be collected by the data acquisition device 560, but may also be received from other devices. It should also be noted that the training device 520 may not train the target model / rule 501 entirely based on the training samples maintained in the database 530, but may also obtain training samples from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0084] The target model / rule 501 obtained by training the training device 520 can be applied to different systems or devices, such as Figure 3 The execution device 510 shown may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle-mounted terminal, etc., or a server, etc.

[0085] Among them, the target model / rule 501 can be used to achieve target tasks, such as driving control in autonomous driving, posture control on a robot, etc.

[0086] Specifically, the training device 520 may transmit the trained model to the execution device 510. The execution device 510 may be the target object mentioned above.

[0087] exist Figure 3 In the embodiment, the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with an external device. The user can input data to the I / O interface 512 through the client device 540, or the execution device 510 can automatically collect the input data.

[0088] Preprocessing module 513 and preprocessing module 514 are used to preprocess the input data received by I / O interface 512. It should be understood that preprocessing module 513 and preprocessing module 514 may be absent or only one preprocessing module may be present. If preprocessing module 513 and preprocessing module 514 are absent, computing module 511 may be used directly to process the input data.

[0089] When the execution device 510 preprocesses the input data, or when the computing module 511 of the execution device 510 performs calculations and other related processing, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 550.

[0090] Finally, the I / O interface 512 provides the processing result to the client device 540 , thereby providing it to the user or performing a control operation based on the processing result.

[0091] exist Figure 3 In the illustrated case, the user can manually input data, and this "manual input data" can be operated through the interface provided by I / O interface 512. In another case, client device 540 can automatically send input data to I / O interface 512. If the automatic transmission of input data by client device 540 requires user authorization, the user can set the corresponding permissions in client device 540. The user can view the results output by execution device 510 on client device 540, and the specific presentation form can be a display, sound, action, etc. Client device 540 can also serve as a data acquisition terminal, collecting input data input into I / O interface 512 and output results from I / O interface 512 as new sample data and storing them in database 530. Of course, collection can also be performed without client device 540, and instead the I / O interface 512 directly stores the input data input into I / O interface 512 and output results from I / O interface 512 as new sample data in database 530.

[0092] It is worth noting that Figure 3This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3 In the embodiment, the data storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the execution device 510 can be deployed in the client device 540.

[0093] From the training side of the model:

[0094] In the embodiment of the present application, the training device 520 can obtain the memory ( Figure 3 Not shown in the figure, the code stored in the training device 520 can be integrated into or deployed separately from the training device 520 to implement the steps related to model training in the embodiments of the present application.

[0095] In an embodiment of the present application, the training device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0096] It should be understood that the training device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to model training provided in the embodiments of the present application can also be implemented by the hardware system in the training device 520 that does not have the function of executing instructions, which is not limited here.

[0097] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0098] (1) Neural Network

[0099] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0100]

[0101] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0102] (2) Deep Neural Networks

[0103] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. Based on the position of different layers in DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0104] (3) Reinforcement learning (RL), also known as reinforcement learning, evaluation learning or enhanced learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how an agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment.

[0105] A common model for reinforcement learning is the standard Markov decision process (MDP). Depending on the given conditions, reinforcement learning can be divided into model-based reinforcement learning (RL) and model-free reinforcement learning (RL), as well as active reinforcement learning (RL) and passive reinforcement learning (RL). Reinforcement learning variants include inverse reinforcement learning, hierarchical reinforcement learning, and reinforcement learning for partially observable systems. Algorithms used to solve reinforcement learning problems can be divided into two categories: policy search algorithms and value function algorithms. Deep learning models can be used in reinforcement learning, forming deep reinforcement learning.

[0106] (4) Loss function

[0107] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0108] (5) Backpropagation algorithm

[0109] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the optimal super-resolution model parameters, such as the weight matrix.

[0110] (6) Reinforcement Learning Model

[0111] Reinforcement learning (RL), also known as reinforcement learning, evaluation learning or enhanced learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how an agent can maximize rewards or achieve specific goals by learning strategies during its interaction with the environment.

[0112] A common model for reinforcement learning is the standard Markov decision process (MDP). Depending on the given conditions, reinforcement learning can be divided into model-based reinforcement learning (RL) and model-free reinforcement learning (RL), as well as active reinforcement learning (RL) and passive reinforcement learning (RL). Reinforcement learning variants include inverse reinforcement learning, hierarchical reinforcement learning, and reinforcement learning for partially observable systems. Algorithms used to solve reinforcement learning problems can be divided into two categories: policy search algorithms and value function algorithms. Deep learning models can be used in reinforcement learning, forming deep reinforcement learning.

[0113] (8) Intelligent Agent

[0114] An agent is a concept in the field of artificial intelligence. Any entity capable of independent thought and interaction with its environment can be abstracted as an agent. The fundamental characteristics of an agent are its ability to respond to changes in its environment, automatically adjusting its behavior and state. Different agents can also interact with other agents based on their own intentions.

[0115] In the typical paradigm of policy optimization, an intelligent agent (e.g., a reinforcement learning model that can predict control policies based on the state of a target object) can actively interact with the environment, receive feedback data from the environment, and use this feedback to continuously improve the agent's policy. However, in practice, the cost of interacting with the environment to obtain data is often very expensive or even unsafe (e.g., in business and healthcare). Therefore, the adverse effects of the data collection process have become a major obstacle to the deployment of policy optimization in practical application problems. On the other hand, a large amount of historical data has been recorded during the execution of existing policies. How to use this data to train intelligent agents is a major opportunity and challenge in the field of policy optimization.

[0116] Because policy optimization is often infeasible in certain applications, offline policy optimization (PO) is often employed. Offline PO involves learning the optimal policy using a fixed dataset derived from the interaction of an existing (unknown) policy with the environment, without allowing the agent to interact with the environment. Since the agent and the environment do not interact, the learning method is called "offline." The policy corresponding to the fixed dataset is called an action policy.

[0117] In an offline setting, there is no interaction between the agent and the environment, and existing datasets only provide partial coverage of the action state space. This is the main difference between Offline PO and standard policy optimization, and also the direct problem it faces. This direct problem makes it difficult for the algorithm to effectively evaluate the value of uncollected actions (also known as out-of-distribution actions, or OOD) during the offline learning phase. Deploying offline learned policies online often yields poor results. Recent work has shown that applying classic RL algorithms (such as Q-learning) to offline datasets exacerbates the collection of data containing OOD actions, which often leads to complete algorithm failure.

[0118] In existing implementations, during the offline strategy optimization of the decision model, due to the lack of training samples (such as the state of the target object), at each step, the state of the target object is predicted by multiple pre-trained machine learning models based on the control strategy obtained by the decision model and the current state of the target object. However, since the state of the target object is predicted, it is often not accurate, so a conservative value function estimation method is used to fit the lower bound. Specifically, the variance of the prediction results of multiple machine learning models can be calculated, and the size of the variance can be controlled by a preset coefficient to achieve the fitting of the lower bound. However, the state obtained by fitting the lower bound only by a fixed coefficient is not accurate, which limits the efficiency and accuracy of reinforcement learning.

[0119] To solve the above problems, refer to Figure 4 , Figure 4 A data processing method according to an embodiment of the present invention is shown in FIG. Figure 4 As shown, a data processing method provided in an embodiment of the present application includes:

[0120] 401. Process a first state through a decision model to obtain first control information; wherein the first state indicates a state of a target object, and the first control information is used as a control information decision model when performing a target task on the target object.

[0121] Among them, the execution subject of step 401 can be a training device (exemplarily, the training device can be a terminal device or a server). For details, please refer to the description in the above embodiment, which will not be repeated here.

[0122] In one possible implementation, the decision model can be a reinforcement learning model, an artificial neural network, a deep neural network, a Bayesian neural network, etc.

[0123] In a possible implementation, the training device may obtain the object of model training (decision model) and the training sample (first state). Optionally, the first state may be a randomly selected state.

[0124] In a possible implementation, the first state is state information related to the robot; the target task is posture control of the robot, and the first control information is posture control information of the robot.

[0125] In one possible implementation, the robot-related status information may include but is not limited to the robot's position, speed, and information related to the scene in which it is located (such as obstacle information). The robot's position and speed may include information such as the status of each joint (position, angle, speed, acceleration, etc.).

[0126] In one possible implementation, the decision model can obtain the robot's posture control information based on the input data. The posture control information may include control information of each joint of the robot. The robot's posture manipulation task can be executed based on the posture control information.

[0127] In a possible implementation, the first state is vehicle-related state information; the target task is automatic driving of the vehicle, and the first control information is driving control information of the vehicle.

[0128] In one possible implementation, vehicle-related status information may include but is not limited to the vehicle's position, speed, and information related to the scene in which it is located (such as information about the driving road, obstacle information, pedestrian information, and information about surrounding vehicles).

[0129] In a possible implementation, the decision model may obtain driving control information of the vehicle based on input data. The driving control information may include information such as the speed, direction, and driving trajectory of the vehicle.

[0130] In a possible implementation, the decision model may be an initialized model or the output of an iteration during the model training process.

[0131] In one possible implementation, during the feedforward process of model training, the first state can be processed by the decision model to obtain first control information. The first control information is used as control information when performing a target task on the target object. For example, if the target task is posture manipulation of a robot, the first control information is the robot's posture control information; or if the target task is autonomous driving of a vehicle, the first control information is the vehicle's driving control information.

[0132] Optionally, in a possible implementation, the first control information may serve as a hard constraint imposed on the target object when executing the target task.

[0133] 402. Process the first state and the first control information through multiple machine learning models to predict multiple second states of the decision model; each second state is a predicted state after controlling the target object in the first state through the first control information.

[0134] 403. Based on the multiple second states, predict the first long-term benefit corresponding to each second state; the multiple first long-term benefits are used to select a target model from the multiple machine learning models; the second state obtained by the target model is used to update the decision model.

[0135] In one possible implementation, the machine learning model can be called a dynamic model. During the training process of the machine learning model, the machine learning model can be enabled to have the ability to predict the next state of the target object based on the state and the control information corresponding to the state.

[0136] In one possible implementation, different machine learning models among the multiple machine learning models have different network structures or different parameter initialization values. Due to the differences in network structures or initialization methods, different machine learning models can predict different results for the same input.

[0137] In one possible implementation, a model set including multiple machine learning models can be trained. Multiple machine learning models constitute the model set. A sampling probability can be set for each machine learning model in the model set. In each training step, the machine learning models in the model set can be probabilistically sampled according to the sampling probability to obtain multiple machine learning models introduced in step 402.

[0138] In one possible implementation, each sampling probability is related to the correlation between the corresponding machine learning model and the processing capabilities of the decision model. For example, the probability set by the machine learning model can be positively correlated with the correlation with the processing capabilities of the decision model, thereby making it easier for machine learning models with a higher correlation with the processing capabilities of the decision model to be sampled, making the subsequent predicted state less conservative, and having a higher probability of obtaining a more accurate prediction result, but at the same time reducing the conservatism of the prediction result.

[0139] In a possible implementation, each of the sampling probabilities is a preset probability value, which can be given by an expert or an experience value, for example.

[0140] In one possible implementation, the first state and the first control information can be processed by multiple machine learning models (for example, the multiple machine learning models obtained by providing probability sampling as mentioned above), each machine learning model can obtain a second state of the decision model, and then predict multiple second states of the decision model; each second state is a predicted state after the target object in the first state is controlled by the first control information.

[0141] In one possible implementation, a target model can be selected from multiple machine learning models based on multiple second states, and the prediction results obtained by the target model are used as the prediction state used when training the decision model.

[0142] Next, we will introduce how to select a target model from multiple machine learning models based on multiple second states.

[0143] Using the second state with higher prediction accuracy as the prediction state for training the decision model is equivalent to using a relatively less conservative prediction result. Using the second state with lower prediction accuracy as the prediction state for training the decision model is equivalent to using a relatively conservative prediction result. Prediction accuracy can be determined based on a profit prediction function. Specifically, the profit prediction function can calculate the long-term profit corresponding to the second state based on the second state obtained by the machine learning model and the first state input to the decision model. Optionally, the profit prediction function can be a Q function.

[0144] In one possible implementation, the second state obtained by each of the multiple machine learning models can predict a long-term benefit. After sorting by size, the second state obtained by the machine learning model with the Nth long-term benefit can be used as the predicted state for training the decision model.

[0145] In one possible implementation, a parameter N can be preset, and the target model is the Nth model with the highest long-term returns among the multiple machine learning models, sorted by size. A larger value for N, that is, a model with a smaller long-term return, results in a more conservative prediction, which may reduce the prediction accuracy but is more likely to be accurate.

[0146] In one possible implementation, in order to make the prediction results more conservative, a machine learning model with a low long-term return ranking can be selected from multiple machine learning models. For example, a model with the Nth-ranked long-term return among multiple machine learning models can be selected, specifically the model with the smallest long-term return among the multiple machine learning models, the model with the second-to-last smallest long-term return, or the model with the third-to-last smallest long-term return, etc.

[0147] In one possible implementation, the first long-term benefit corresponding to each second state can be predicted based on the multiple second states through the Q function, and the Q function can be updated. Specifically, the second long-term benefit corresponding to the first state can be predicted based on the first state through the Q function, and the Q function can be updated based on the difference between the first long-term benefit and the second long-term benefit.

[0148] Next, the data processing method in the embodiment of the present application is introduced with reference to a specific process:

[0149] In one possible implementation, the a priori parameters of the setting algorithm may be given, specifically including the probability distribution P_T obeyed by the a priori model (that is, the machine learning model introduced in the embodiment of the present application), the number of a priori models N, the a priori conservative control value k (that is, the parameter N introduced above), the number of core process iterations M, etc. The specific process is as follows:

[0150] In one possible implementation, the parameters theta of the Q function and phi of the policy function (i.e., the decision model to be trained) can be randomly initialized. A target Q function theta' and a reference policy phi' are set, with the same parameters as the Q function and policy function. C states are randomly sampled from the dataset and used as the initial states of C alternating Markov gama (AMG). Based on the existing states, actions are sampled according to the current optimal policy. A prior model is sampled based on the belief distribution and applied to the current state and action to obtain offline data. Based on the data from the previous step and the samples from the original dataset, the Q function and policy function are updated with a one-step gradient iteration to reduce the objective function associated with them. The prior model that results in the current optimal policy having the kth smallest objective function is selected with probability 1-epsilon; otherwise, a random sample is sampled from the prior model with probability epsilon. The state is updated based on the prior model sampled in the previous step. The parameters of the reference policy and target Q function are updated using a sliding average. Repeat the above steps M times until convergence and return to the reference policy.

[0151] For example, you can refer to Figure 5 , Figure 5 The flowchart of a data processing method includes the following steps 1 to 9.

[0152] 1) Randomly initialize the Q function parameters theta and the policy function parameters phi. At the same time, set a target Q function theta' and a reference policy phi', whose parameters are the same as the Q function and policy function.

[0153] 2) Randomly extract C states from the dataset and use them as the initial states of C AMGs.

[0154] 3) Based on the existing state, sample actions according to the current optimal strategy.

[0155] 4) Sampling the prior model based on the belief distribution and applying it to the current state and action to obtain offline data.

[0156] 5) Based on the data from the previous step and the sampling in the original data set, the Q function and the policy function are updated with a one-step gradient iteration to reduce the objective function related to the Q function and the policy function.

[0157] 6) Select the prior model with a probability of 1-epsilon that makes the current optimal strategy have the kth smallest objective function; otherwise, randomly select from the prior model with a probability of epsilon.

[0158] 7) Update the state based on the prior model sampled in the previous step.

[0159] 8) Update the parameters of the reference strategy and target Q function in a sliding average manner.

[0160] 9) Repeat steps 3)–8) M times until convergence and return to the reference policy.

[0161] An embodiment of the present application provides a data processing method, which includes: processing a first state through a decision model to obtain first control information; wherein, the first state indicates the state of a target object, and the first control information is used as control information when performing a target task on the target object; processing the first state and the first control information through multiple machine learning models to predict multiple second states of the decision model; each second state is a predicted state after controlling the target object in the first state through the first control information; based on the multiple second states, predicting the first long-term benefit corresponding to each second state; the multiple first long-term benefits are used to select a target model from the multiple machine learning models; the second state obtained by the target model is used to update the decision model.

[0162] In existing technologies, the uncertainty distribution of machine learning model outputs and fixed coefficients are used to control the size of the distribution. To ensure that the predicted state does not make serious errors, the coefficient settings are often overly conservative, resulting in poor performance of the trained model. Furthermore, the impact of machine learning model uncertainty on long-term returns is not precisely described. This inaccuracy is reflected in the problem of incorrect numerical results due to inconsistent dimensions and overly conservative approaches in low model confidence levels. Furthermore, the existing technology generally uses fixed coefficients to control the size of uncertainty, lacking a global perspective on the entire sequential decision-making process. This lack of global perspective has a negative impact on policy optimization, and it cannot dynamically update the posterior probability based on the training results.

[0163] In an embodiment of the present application, the machine learning model used as the prediction state is selected based on the long-term benefits of the second state obtained by the machine learning model. This can characterize the uncertainty of the prediction state in real time, and adaptively adjust the selection of the machine learning model. It can more accurately select a prediction state with a certain degree of conservatism, thereby improving the performance of the decision-making model.

[0164] In addition, by setting the sampling probability corresponding to the machine learning model and further selecting the machine learning model based on long-term returns, the conservatism can be flexibly adjusted to achieve a balance between conservative or aggressive strategies.

[0165] This application addresses the existing problems of Robust MDP and provides a new characterization method for Offline PO based on the alternating Markov game (AMG) framework. This characterization method can not only make full use of the prior information of the model, but also reasonably adjust the posterior probability of the model based on the collected data distribution, and is directly used to deal with the problem of single-step policy optimization. In addition, based on the AMG framework, a new objective function for offline policy optimization is proposed. The objective function contains two adjustment parameters, through which the conservatism can be flexibly adjusted, allowing a balance to be achieved from conservative or aggressive strategies.

[0166] Next, a data processing method in an embodiment of the present application is described with reference to a specific example:

[0167] This embodiment uses an example in which P_T has four possible uniform distributions, the number of prior models N=3, the prior conservative control value k=2, and the number of core process iterations M=1 to illustrate the algorithm.

[0168] About model initialization:

[0169] Assuming that the initial state and action space are both finite and discrete, there are only two different states (0 / 1) and actions (0 / 1). Therefore, the Q function and policy function can be expressed as two tables, each with 4 parameters to be estimated:

[0170] Q(s,a) Action 0 Action 1 State 0 Theta1 Theta2 State 1 Theta3 Theta4

[0171] pi(a|s) Action 0 Action 1 State 0 Phi1 Phi2 State 1 Phi3 Phi4

[0172] Set the target Q function (denoted as TQ) as the Q function and the reference policy function (denoted as Rpi) as the policy function:

[0173] TQ(s,a) Action 0 Action 1 State 0 Theta'1 Theta'2 State 1 Theta'3 Theta'4

[0174] Rpi(a|s) Action 0 Action 1 State 0 Phi'1 Phi'2 State 1 Phi'3 Phi'4

[0175] We can randomly extract a state s from the data set. Since there are only two possible states in the data set, {0, 1}, we can assume that s = 0. Calculate the product of Rpi and the exponential Q function to obtain the following probability table:

[0176] Sampling probability Action 0 Action 1 State 0 Phi'1*exp(10*Theta1) Phi'2*exp(10*Theta2)

[0177] Based on the probability table, two actions are sampled, and it can be assumed that the sampled action is a=1.

[0178] Assume that the candidate model set is {tau1, tau2, tau3, tau4}, and N = 3 models are randomly extracted from it. The probabilities of various possible extraction results are as follows:

[0179]

[0180] Let us assume that the sampling result is Tau = {tau1, tau2, tau3}.

[0181] Extract the data (s,a,s') = (0,1,0) from the dataset D and combine it with the data (s,a,Tau) = (0,1,{tau1,tau2,tau3}) to update the values of the Q function table: Theta1,Theta2,Theta3,Theta4. Similarly, we also update the values of the policy function table: Phi1,Phi2,Phi3,Phi4.

[0182] Given s and a, set the transition probability to tau1, and sample the next state s', assuming s' = 1. Based on the probability of the reference policy being in state 1, sample the next action a'.

[0183] Rpi(a|s) Action 0 Action 1 State 1 Phi'3 Phi'4

[0184] It can be assumed that the sampled result is a' = 1. Next, v1 = exp{10*Theta'3} is calculated.

[0185] Similarly, according to the above process, for tau2 and tau3 in Tau, we calculate the values v2 and v3 respectively. From v1, v2, and v3, we select the second-ranked value, assuming it is v2. Then the probability model extracted in this step is tau2.

[0186] Based on the sampled probability transition model tau2, the current state s = 0, and the current action a = 1, we extract the next state s'. Assuming s' = 1, we use the value of s' to update the value of s to obtain the new value of the current state.

[0187] The new reference strategy is updated in a weighted manner. Here, weighting means taking into account both the latest strategy and the reference strategy, with the specific weights being omega1 and 1-omega1 respectively. Therefore, the updated reference strategy table is:

[0188]

[0189] At this point, the iteration step is completed.

[0190] Reference Figure 6 , Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device 600 includes:

[0191] A processing module 601 is configured to process a first state through a decision model to obtain first control information, wherein the first state indicates a state of a target object, and the first control information is used as control information when performing a target task on the target object;

[0192] Processing the first state and the first control information through multiple machine learning models to predict multiple second states of the decision model; each second state is a predicted state after controlling the target object in the first state through the first control information;

[0193] The specific description of the processing module 601 can refer to the description of step 401 and step 402 in the above embodiment, which will not be repeated here.

[0194] The profit prediction module 602 is used to predict the first long-term profit corresponding to each of the multiple second states based on the multiple second states; the multiple first long-term profits are used to select a target model from the multiple machine learning models; the second state obtained by the target model is used to update the decision model.

[0195] The specific description of the revenue prediction module 602 can refer to the description of step 403 in the above embodiment, which will not be repeated here.

[0196] In one possible implementation,

[0197] The target object is a robot; the target task is posture control of the robot, and the first control information is posture control information of the robot; or,

[0198] The target object is a vehicle; the target task is automatic driving of the vehicle, and the first control information is driving control information of the vehicle.

[0199] In one possible implementation, the target model is the Nth long-term benefit model among the multiple machine learning models sorted by size, where N is a preset parameter.

[0200] In one possible implementation, the model with the Nth long-term benefit is specifically the model with the smallest long-term benefit, the model with the second smallest long-term benefit, or the model with the third smallest long-term benefit among the multiple machine learning models.

[0201] In one possible implementation, the multiple machine learning models are sampled from a model set through probability sampling, wherein each machine learning model in the model set corresponds to a sampling probability.

[0202] In one possible implementation, each of the sampling probabilities is related to a correlation between the corresponding machine learning model and the processing capabilities of the decision model.

[0203] In a possible implementation, each of the sampling probabilities is a preset probability value.

[0204] In one possible implementation, different machine learning models among the multiple machine learning models have different network structures or different parameter initialization values.

[0205] In a possible implementation, the revenue prediction module is further configured to:

[0206] According to the first state, using a profit prediction function, predicting a second long-term profit corresponding to the first state;

[0207] The step of predicting, based on the plurality of second states, a long-term benefit corresponding to each second state includes:

[0208] The revenue prediction module is specifically used to:

[0209] The profit prediction function is updated according to the difference between the first long-term profit and the second long-term profit.

[0210] Next, we will introduce an execution device provided by the embodiment of the present application. Figure 7 , Figure 7 This is a structural diagram of an execution device provided in an embodiment of the present application. The execution device 700 can be specifically manifested as a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which is not limited here. Specifically, the execution device 700 includes: a receiver 701, a transmitter 702, a processor 703 and a memory 704 (wherein the number of processors 703 in the execution device 700 can be one or more, Figure 7 (taking one processor as an example), the processor 703 may include an application processor 7031 and a communication processor 7032. In some embodiments of the present application, the receiver 701, the transmitter 702, the processor 703 and the memory 704 may be connected via a bus or other means.

[0211] The memory 704 may include a read-only memory and a random access memory, and provides instructions and data to the processor 703. A portion of the memory 704 may also include non-volatile random access memory (NVRAM). The memory 704 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0212] Processor 703 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0213] The methods disclosed in the above embodiments of the present application can be applied to or implemented by the processor 703. The processor 703 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 703. The above processor 703 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 703 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 704, and processor 703 reads the information in memory 704 and performs the steps of the above method in conjunction with its hardware.

[0214] Receiver 701 can be used to receive input digital or character information and generate signal input related to executing relevant settings and function control of the device. Transmitter 702 can be used to output digital or character information; transmitter 702 can also be used to send instructions to the disk group to modify the data in the disk group.

[0215] In one embodiment of the present application, the processor 703 is configured to execute Figure 4 The steps of the model obtained by the data processing method in the corresponding embodiment.

[0216] The present application also provides a server. Figure 8 , Figure 8This is a structural diagram of a server provided in an embodiment of the present application. Specifically, the server 800 is implemented by one or more servers. The server 800 may have relatively large differences due to different configurations or performances. It may include one or more central processing units (CPUs) 88 (for example, one or more processors) and memory 832, and one or more storage media 830 (for example, one or more mass storage devices) for storing application programs 842 or data 844. Among them, the memory 832 and the storage medium 830 can be temporary storage or permanent storage. The program stored in the storage medium 830 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 88 can be configured to communicate with the storage medium 830 to execute a series of instruction operations in the storage medium 830 on the server 800.

[0217] The server 800 may also include one or more power supplies 86, one or more wired or wireless network interfaces 850, one or more input and output interfaces 858; or one or more operating systems 841, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0218] In the embodiment of the present application, the central processing unit 88 is used to execute Figure 4 The steps of the data processing method in the corresponding embodiment.

[0219] An embodiment of the present application also provides a computer program product including computer-readable instructions, which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0220] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0221] The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the model training method described in the above embodiment, or so that the chip in the training device executes the steps related to model training in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0222] For details, please refer to Figure 9 , Figure 9 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 900. NPU 900 is mounted on the host CPU as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 903, which is controlled by controller 904 to extract matrix data from memory and perform multiplication operations.

[0223] In some implementations, the arithmetic circuit 903 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 903 is a two-dimensional systolic array. The arithmetic circuit 903 may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 903 is a general-purpose matrix processor.

[0224] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 902 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 901 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 908.

[0225] Unified memory 906 is used to store input and output data. Weight data is directly transferred to weight memory 902 through the Direct Memory Access Controller (DMAC) 905. Input data is also transferred to unified memory 906 through the DMAC.

[0226] BIU stands for Bus Interface Unit, i.e., bus interface unit 99 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 909 .

[0227] The bus interface unit 99 (BIU) is used for the instruction fetch memory 909 to obtain instructions from the external memory, and is also used for the storage unit access controller 905 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0228] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 906 or transfer weight data to the weight memory 902 or transfer input data to the input memory 901.

[0229] The vector calculation unit 907 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0230] In some implementations, the vector calculation unit 907 can store the processed output vector in the unified memory 906. For example, the vector calculation unit 907 can apply a linear function or a nonlinear function to the output of the operation circuit 903, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values to generate an activation value. In some implementations, the vector calculation unit 907 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 903, for example, for use in subsequent layers in a neural network.

[0231] An instruction fetch buffer 909 connected to the controller 904 is used to store instructions used by the controller 904;

[0232] Unified memory 906, input memory 901, weight memory 902, and instruction fetch memory 909 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0233] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0234] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0235] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0236] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0237] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, computer, training device or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive (SSD)).

Claims

1. A data processing method, characterized in that: The method comprises: Processing a first state through a decision model to obtain first control information; wherein the first state indicates a state of a target object, the first control information is used as control information when performing a target task on the target object, and the first state is a randomly selected state; Processing the first state and the first control information through multiple machine learning models to predict multiple second states of the decision model; each second state is a predicted state after the target object in the first state is controlled by the first control information, and the multiple machine learning models are sampled from a model set through probability sampling, wherein each machine learning model in the model set corresponds to a sampling probability; Based on the multiple second states, the first long-term benefit corresponding to each second state is predicted; the multiple first long-term benefits are used to select a target model from the multiple machine learning models; the second state obtained by the target model is used to update the decision model.

2. The method according to claim 1, characterized in that The target object is a robot; the target task is posture control of the robot, and the first control information is posture control information of the robot; or, The target object is a vehicle; the target task is automatic driving of the vehicle, and the first control information is driving control information of the vehicle.

3. The method according to claim 1 or 2, characterized in that The target model is the Nth long-term benefit model among the multiple machine learning models sorted by size, where N is a preset parameter.

4. The method according to claim 3, characterized in that The model with the Nth long-term return is specifically the model with the smallest long-term return, the model with the second smallest long-term return, or the model with the third smallest long-term return among the multiple machine learning models.

5. The method according to claim 1, wherein Each of the sampling probabilities is related to the correlation between the corresponding machine learning model and the processing capabilities of the decision model.

6. The method according to claim 1, characterized in that Each of the sampling probabilities is a preset probability value.

7. The method according to any one of claims 1 to 2, characterized in that: Different machine learning models among the multiple machine learning models have different network structures or different parameter initialization values.

8. The method according to any one of claims 1 to 2, characterized in that: The method further comprises: According to the first state, predicting a second long-term benefit corresponding to the first state by using a benefit evaluation function; The step of predicting, based on the plurality of second states, a long-term benefit corresponding to each second state includes: According to the plurality of second states, predicting the first long-term benefit corresponding to each second state by using the benefit evaluation function; The benefit evaluation function is updated according to the difference between the first long-term benefit and the second long-term benefit.

9. A data processing device, characterized in that: The device comprises: a processing module, configured to process a first state through a decision model to obtain first control information; wherein the first state indicates a state of a target object, the first control information is used as control information when performing a target task on the target object, and the first state is a randomly selected state; Processing the first state and the first control information through multiple machine learning models to predict multiple second states of the decision model; each second state is a predicted state after the target object in the first state is controlled by the first control information, and the multiple machine learning models are sampled from a model set through probability sampling, wherein each machine learning model in the model set corresponds to a sampling probability; A profit prediction module is used to predict the first long-term profit corresponding to each of the multiple second states based on the multiple second states; the multiple first long-term profits are used to select a target model from the multiple machine learning models; and the second state obtained by the target model is used to update the decision model.

10. The device according to claim 9, characterized in that The target object is a robot; the target task is posture control of the robot, and the first control information is posture control information of the robot; or, The target object is a vehicle; the target task is automatic driving of the vehicle, and the first control information is driving control information of the vehicle.

11. The device according to claim 9 or 10, characterized in that The target model is the Nth long-term benefit model among the multiple machine learning models sorted by size, where N is a preset parameter.

12. The device according to claim 11, characterized in that The model with the Nth long-term return is specifically the model with the smallest long-term return, the model with the second smallest long-term return, or the model with the third smallest long-term return among the multiple machine learning models.

13. The device according to claim 9, characterized in that Each of the sampling probabilities is related to the correlation between the corresponding machine learning model and the processing capabilities of the decision model.

14. The device according to claim 9, characterized in that Each of the sampling probabilities is a preset probability value.

15. The device according to any one of claims 9 to 10, characterized in that Different machine learning models among the multiple machine learning models have different network structures or different parameter initialization values.

16. The device according to any one of claims 9 to 10, characterized in that The revenue prediction module is further used to: According to the first state, predicting a second long-term benefit corresponding to the first state by using a benefit evaluation function; The step of predicting, based on the plurality of second states, a long-term benefit corresponding to each second state includes: The revenue prediction module is specifically used to: The benefit evaluation function is updated according to the difference between the first long-term benefit and the second long-term benefit.

17. A data processing device, characterized in that: The device includes a memory and a processor; the memory stores codes, and the processor is configured to obtain the codes and execute the method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that The method comprises computer-readable instructions, which, when executed on a computer device, cause the computer device to execute the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on a computer device, cause the computer device to execute the method according to any one of claims 1 to 8.

Images