A data processing method and related device

Abstract

The application discloses a data processing method, and relates to the field of artificial intelligence, and the method comprises the following steps: acquiring first prompt information; the first prompt comprises information of a first task; obtaining first control information through a pre-training model according to the information of the first task; updating the pre-training model according to a first execution result obtained by a target object based on the first control information when the target object executes the first task, to obtain an updated pre-training model; acquiring second prompt information; the second prompt comprises information of a second task; the execution complexity of the second task is higher than that of the first task; obtaining second control information through the updated pre-training model according to the information of the second task and information of the target object; and updating the updated pre-training model according to a second execution result obtained by the target object based on the second control information when the target object executes the second task. The application can start learning from simple sub-skills, gradually evolve learning from simple to complex task skills, and enable the model to learn complex skills.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a data processing method and related equipment. Background Technology

[0002] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that simulate, extend, and expand human intelligence through digital computers or machines controlled by digital computers, enabling them to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] Object manipulation technologies based on natural language processing models (such as robot posture control or autonomous driving of vehicles) can be widely applied in scenarios such as robot posture control or autonomous driving of vehicles. Taking robots as an example, robot task skill learning is a challenging task. Existing robot skill learning in the field of natural language manipulation often requires the collection of large amounts of natural language data and the introduction of a large amount of human-in-the-loop feedback information, resulting in low efficiency, high cost, and slow learning. Therefore, it is hoped that the powerful reasoning ability of the basic model can be used to quickly learn the robot's complex decision-making skills.

[0004] In existing technologies, a pre-trained natural language base model is used to manually define tasks through a prompt. After receiving prompt information such as task constraints, the base model outputs corresponding decision behavior (or control information, such as instructions or scripts). When the model's output behavior is incorrect, humans use natural language feedback to allow the base model to correct its decision output, iterating repeatedly until the decision output is completely correct.

[0005] However, as the complexity of the tasks to be accomplished increases, existing technologies suffer from high skill learning costs, long learning cycles, and difficulty in learning complex skills. Summary of the Invention

[0006] This application provides a data processing method that can improve the learning efficiency of the model. Starting with simple sub-skills, the model gradually evolves and learns task skills from simple to complex, enabling it to learn more complex skills.

[0007] In a first aspect, this application provides a data processing method, the method comprising: acquiring first prompt information; the first prompt including information about a first task and information about a target object, wherein the first task is a task specified for the target object to perform; obtaining first control information through a pre-trained model based on the information about the first task and the information about the target object; the first control information being used as control information for the target object when performing the first task; updating the pre-trained model based on a first execution result obtained by the target object performing the first task based on the first control information, thereby obtaining an updated pre-trained model; acquiring second prompt information; the second prompt including information about a second task; the execution complexity of the second task being higher than that of the first task; the second task being a task specified for the target object to perform; obtaining second control information through the updated pre-trained model based on the information about the second task and the information about the target object; the second control information being used as control information for the target object when performing the second task; and updating the updated pre-trained model based on a second execution result obtained by the target object performing the second task based on the second control information.

[0008] By using the above methods, the learning efficiency of the model can be improved. Starting with simple sub-skills and gradually evolving the learning from simple to complex task skills, the model can learn more complex skills.

[0009] In one possible implementation, the first task is a subtask that needs to be executed when performing the second task. The first task has lower complexity than the second task; that is, the first task is a basic task, and the second task is a more advanced task. For example, the first task could be a grasping action, and the second task could be a task of tidying up a desktop; or the first task could be a turning task, and the second task could be a task of obstacle avoidance. Optionally, the advanced task can be implemented by executing a combination of multiple basic tasks.

[0010] In one possible implementation, the second prompt information further includes: instruction information that instructs the target object to perform the second task through a combination of control information for multiple learned tasks, including the first task.

[0011] In other words, when defining a task, a prompt can be used to inform the pre-trained model that it can continuously combine existing skills to autonomously form new skills during the learning process. That is, when learning more complex skills, new and more complex skills will evolve based on previously learned skills. This approach improves the model's learning efficiency, and by starting with simple sub-skills and gradually evolving the learning of task skills from simple to complex, the model can learn more sophisticated skills.

[0012] In one possible implementation, the first prompt information or the second prompt information further includes: information about a reward function; the information about the reward function indicates the numerical range of the reward function and / or the relationship between the execution result and the numerical value of the reward function; updating the pre-trained model based on the first execution result obtained by the target object performing the first task based on the first control information includes: obtaining a first reward value based on the first execution result obtained by the target object performing the first task based on the first control information and the information about the reward function, and updating the pre-trained model based on the first reward value; or, updating the updated pre-trained model based on the second execution result obtained by the target object performing the second task based on the second control information includes: obtaining a second reward value based on the second execution result obtained by the target object performing the second task based on the second control information and the information about the reward function, and updating the updated pre-trained model based on the second reward value.

[0013] In one possible implementation, the first prompt or the second prompt further includes: information about the environment in which the target object is located; obtaining the first control information through a pre-trained model based on the information of the first task and the information of the target object includes: obtaining the first control information through a pre-trained model based on the information of the first task, the information of the target object, and the information of the environment; or, obtaining the second control information through the updated pre-trained model based on the information of the second task and the information of the target object includes: obtaining the second control information through the updated pre-trained model based on the information of the first task, the information of the target object, and the information of the environment.

[0014] In one possible implementation, updating the pre-trained model based on the first execution result obtained by the target object performing the first task based on the first control information to obtain an updated pre-trained model includes: determining a first reward value based on a first similarity between the first execution result obtained by the target object performing the first task based on the first control information and the information of the first task, and updating the pre-trained model based on the first reward value to obtain an updated pre-trained model; or, updating the updated pre-trained model based on the second execution result obtained by the target object performing the second task based on the second control information includes: determining a second reward value based on a second similarity between the second execution result obtained by the target object performing the second task based on the second control information and the information of the second task, and updating the updated pre-trained model based on the second reward value.

[0015] In one possible implementation, the information of the first task includes textual description information related to the first task and an execution result image related to the first task; the first similarity is determined based on the similarity between the first execution result and the textual description information related to the first task, and the similarity between the first execution result and the execution result image related to the first task; or, the information of the second task includes textual description information related to the second task and an execution result image related to the second task; the second similarity is determined based on the similarity between the second execution result and the textual description information related to the second task, and the similarity between the second execution result and the execution result image related to the second task.

[0016] In one possible implementation, the information of the target object includes: interface information of the manipulation components on the target object, and / or interface information of the sensing components on the target object.

[0017] In one possible implementation, the pre-trained model is chatGPT, GPT4, or chatGLM.

[0018] In one possible implementation, the target object is a robot; the first task and the second task are posture control of the robot; or, the target object is a vehicle; the first task and the second task are autonomous driving of the vehicle.

[0019] Secondly, this application provides a data processing method, the method comprising:

[0020] Obtain target prompt information; the target prompt includes information about the target task.

[0021] Based on the information of the target task, target control information is obtained through an updated pre-trained model obtained by any of the methods in the first aspect; the target control information is used as control information for the target object when performing the target task.

[0022] Thirdly, this application provides a data processing apparatus, the apparatus comprising:

[0023] The acquisition module is used to acquire first prompt information; the first prompt includes information about a first task and information about a target object, wherein the first task is a task specified to be performed by the target object;

[0024] The processing module is configured to obtain first control information based on the information of the first task and the information of the target object through a pre-trained model; the first control information is used as control information for the target object when performing the first task.

[0025] Based on the first execution result obtained by the target object performing the first task based on the first control information, the pre-trained model is updated to obtain the updated pre-trained model;

[0026] The acquisition module is further configured to acquire second prompt information; the second prompt includes information about a second task; the execution complexity of the second task is higher than that of the first task; the second task is a task specified for the target object to be executed.

[0027] The processing module is further configured to obtain second control information based on the information of the second task and the information of the target object through the updated pre-trained model; the second control information is used as control information for the target object when performing the second task.

[0028] The updated pre-trained model is updated based on the second execution result obtained by the target object performing the second task based on the second control information.

[0029] In one possible implementation, the first task is a subtask that needs to be executed when the second task is performed.

[0030] In one possible implementation, the second prompt message also includes:

[0031] The instruction information instructs the target object to perform the second task by a combination of control information for multiple learned tasks, including the first task.

[0032] In one possible implementation, the first prompt information or the second prompt information further includes: information about the reward function; the information about the reward function indicates the numerical range of the reward function and / or the relationship between the execution result and the numerical value of the reward function;

[0033] The processing module is specifically configured to obtain a first reward value based on the first execution result obtained by the target object performing the first task based on the first control information and the information of the reward function, and update the pre-trained model based on the first reward value; or,

[0034] The processing module is specifically used for:

[0035] Based on the second execution result obtained by the target object performing the second task based on the second control information and the information of the reward function, a second reward value is obtained, and the updated pre-trained model is updated based on the second reward value.

[0036] In one possible implementation, the first prompt or the second prompt may further include: information about the environment in which the target object is located;

[0037] The processing module is specifically used to: obtain first control information based on the information of the first task, the information of the target object, and the information of the environment through a pre-trained model; or,

[0038] The processing module is specifically used to: obtain second control information based on the information of the first task, the information of the target object, and the information of the environment through the updated pre-trained model.

[0039] In one possible implementation, the processing module is specifically configured to: determine a first reward value based on a first similarity between the first execution result obtained by the target object performing the first task based on the first control information and the information of the first task; and update the pre-trained model based on the first reward value to obtain an updated pre-trained model; or...

[0040] The processing module is specifically used to: determine a second reward value based on the second similarity between the second execution result obtained by the target object performing the second task based on the second control information and the information of the second task, and update the updated pre-trained model based on the second reward value.

[0041] In one possible implementation, the information of the first task includes textual description information related to the first task and an execution result image related to the first task; the first similarity is determined based on the similarity between the first execution result and the textual description information related to the first task, and the similarity between the first execution result and the execution result image related to the first task; or,

[0042] The information of the second task includes textual description information related to the second task and execution result images related to the second task; the second similarity is determined based on the similarity between the second execution result and the textual description information related to the second task, and the similarity between the second execution result and the execution result images related to the second task.

[0043] In one possible implementation, the information of the target object includes: interface information of the manipulation components on the target object, and / or interface information of the sensing components on the target object.

[0044] In one possible implementation, the pre-trained model is chatGPT, GPT4, or chatGLM.

[0045] In one possible implementation, the target object is a robot; the first task and the second task are the robot's posture control; or,

[0046] The target object is a vehicle; the first task and the second task are the autonomous driving of the vehicle.

[0047] Fourthly, this application provides a data processing apparatus, the apparatus comprising:

[0048] The acquisition module is used to acquire target prompt information; the target prompt includes information about the target task.

[0049] The processing module is used to obtain target control information based on the information of the target task, through an updated pre-trained model obtained by any of the methods in the first aspect; the target control information is used as control information for the target object when performing the target task.

[0050] Fifthly, embodiments of this application provide a data processing apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect above and any of its optional methods.

[0051] In a sixth aspect, embodiments of this application provide a data processing apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the second aspect above and any of its optional methods.

[0052] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any optional methods thereof, or the methods described in the second aspect and any optional methods thereof.

[0053] Eighthly, embodiments of this application provide a computer program product including instructions that, when run on a computer, cause the computer to perform the first aspect and any optional method thereof, or the second aspect and any optional method thereof.

[0054] Ninthly, this application provides a chip system including a processor for supporting a data processing device in implementing some or all of the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of an application architecture;

[0056] Figure 2 This is a schematic diagram of an application architecture;

[0057] Figure 3A This is a schematic diagram of an application architecture;

[0058] Figure 3B This is a schematic diagram of an application architecture;

[0059] Figure 3C This is a schematic diagram of an application architecture;

[0060] Figure 4 This is an illustration of an embodiment of a data processing method provided in this application.

[0061] Figure 5 This is a schematic diagram of the data processing method provided in the embodiments of this application;

[0062] Figure 6 This is an illustration of an embodiment of a data processing method provided in this application.

[0063] Figure 7This application provides an embodiment of a data processing apparatus.

[0064] Figure 8 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0065] Figure 9 This is a schematic diagram of a server structure provided in an embodiment of this application;

[0066] Figure 10 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0067] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0068] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0069] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0070] The terms “substantially,” “about,” and similar terms used herein are used as approximations rather than as terms of degree, and are intended to take into account the inherent biases of measurements or calculations known to those skilled in the art. Furthermore, the use of “may” in describing embodiments of the invention refers to “one or more possible embodiments.” The terms “use,” “using,” and “used” used herein are to be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Additionally, the term “exemplary” is intended to refer to an instance or illustration.

[0071] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0072] (1) Infrastructure

[0073] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0074] (2) Data

[0075] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0076] (3) Data processing

[0077] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0078] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0079] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0080] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0081] (4) General ability

[0082] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0083] (5) Smart Products and Industry Applications

[0084] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0085] With the development of artificial intelligence, many tasks that previously required human intervention are gradually being replaced by smart terminals. These smart terminals need to be equipped with the skills required to complete these tasks, as well as neural networks specific to those tasks, to enable them to perform specific tasks. Specifically, this can be applied to mobile smart terminals. For example, in the field of autonomous driving, driving operations that were originally performed by humans can be performed by smart cars. Therefore, smart cars need to be equipped with a large number of driving skills and neural networks specific to those skills. As another example, in the freight industry, handling operations that were originally performed by humans can be performed by handling robots. Therefore, handling robots need to be equipped with a large number of handling skills and neural networks specific to those skills. It can also be applied to smart terminals that do not have mobile operation capabilities. For example, on a parts processing assembly line, the parts grasping operation that was originally done by humans can be done by a smart robotic arm. In this case, the smart robotic arm needs to be equipped with grasping skills and neural networks for grasping skills. Different grasping skills can have different grasping angles and displacements of the smart robotic arm. As another example, in the field of automatic cooking, the cooking operation that was originally done by humans can be done by a smart robotic arm. In this case, the smart robotic arm needs to be equipped with raw material grasping skills, stir-frying skills, and other cooking skills, as well as neural networks for cooking skills, etc. Other application scenarios will not be exhaustively listed here.

[0086] To better understand the solutions of the embodiments of this application, the following will first combine... Figure 2 and Figure 3A A brief introduction to the possible implementation architectures of the embodiments of this application is provided.

[0087] Figure 2This is a schematic diagram of a computing system for performing model training in an embodiment of this application. The computing system includes a terminal device 102 (exemplary, but not necessarily including the terminal device 102) and a server 130 (also referred to as a central node) coupled via a network communication. The terminal device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0088] Terminal device 102 may include processor 112 and memory 114. Processor 112 may be any suitable processing device (e.g., processor core, microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), controller, microcontroller, etc.). Memory 114 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). Memory 114 may store data 116 and instructions 118 executed by processor 112 to cause terminal device 102 to perform operations.

[0089] In some implementations, memory 114 may store one or more models 120. For example, model 120 may be or may additionally include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks.

[0090] In some implementations, one or more models 120 may be received from server 130 via a network, stored in memory 114, and then used or otherwise implemented by one or more processors 112.

[0091] Terminal device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). Touch-sensitive components can be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices from which the user can provide input.

[0092] The terminal device 102 may also include a communication interface 123. The terminal device 102 can communicate with the server 130 through the communication interface 123. The server 130 may include a communication interface 133. The terminal device 102 can communicate with the server 130 through the communication interface 123 and the communication interface 133 of the server 130, thereby realizing data interaction between the terminal device 102 and the server 130.

[0093] Server 130 may include processor 132 and memory 134. Processor 132 may be any suitable processing device (e.g., processor core, microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), controller, microcontroller, etc.). Memory 134 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). Memory 134 may store data 136 and instructions 138 executed by processor 132 to cause server 130 to perform operations.

[0094] As described above, memory 134 can store one or more machine learning models 140. For example, model 140 can be, or may additionally include, various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0095] It should be understood that the data processing methods in the embodiments of this application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and the server is not limited to... Figure 2 The processor and memory architecture shown is illustrated below. Figure 3AThe system architecture provided in the embodiments of this application will be described in detail.

[0096] Figure 3A This is a schematic diagram of the system architecture provided for an embodiment of this application. For example... Figure 3A As shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0097] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, while the preprocessing modules 513 and 514 are optional.

[0098] Data acquisition device 560 is used to collect training samples. Training samples may include prompt information, such as information about the target task (e.g., autonomous driving, robot posture control), information about the target object, and information about the environment in which the target object is located (e.g., simulation environment). After collecting the training samples, data acquisition device 560 stores these training samples in database 530.

[0099] The training device 520 can maintain training samples in the database 530 and obtain the target model / rule 501 (e.g., the updated pre-trained model in the embodiments of this application) based on the neural network to be trained (e.g., the pre-trained model in the embodiments of this application).

[0100] It should be noted that in practical applications, the training samples maintained in database 530 may not all come from the data acquisition device 560; they may also be received from other devices. Furthermore, it should be noted that training device 520 may not necessarily train the target model / rule 501 entirely based on the training samples maintained in database 530; it may also obtain training samples from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0101] The target model / rule 501 trained using training device 520 can be applied to different systems or devices, such as... Figure 3A The execution device 510 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server, etc.

[0102] Among them, the target model / rule 501 can be used to achieve tasks, such as driving control in autonomous driving and posture control on robots.

[0103] Specifically, the training device 520 can transfer the trained model to the execution device 510. The execution device 510 can be the target object mentioned above.

[0104] exist Figure 3A In the execution device 510, an input / output (I / O) interface 512 is configured for data interaction with external devices. Users can input data to the I / O interface 512 through the client device 540, or the execution device 510 can automatically collect input data.

[0105] Preprocessing modules 513 and 514 are used to preprocess the input data received from the I / O interface 512. It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can be used directly to process the input data.

[0106] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0107] Finally, the I / O interface 512 provides the processing result to the client device 540, which then provides it to the user or performs control operations based on the processing result.

[0108] exist Figure 3A In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various ways such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0109] It is worth noting that, Figure 3A This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3A In this context, the data storage system 550 is an external storage device relative to the execution device 510. However, in other cases, the data storage system 550 may also be placed within the execution device 510. It should be understood that the aforementioned execution device 510 may be deployed within the client device 540.

[0110] From the training side of the model:

[0111] In this embodiment of the application, the training device 520 can access the memory ( Figure 3A (Not shown in the diagram, but can be integrated into the training device 520 or deployed separately from the training device 520) The code stored in the diagram can be used to implement the steps related to model training in the embodiments of this application.

[0112] In this embodiment of the application, the training device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0113] It should be understood that the training device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps related to the training of the neutralization model provided in the embodiments of this application can also be implemented by the hardware system in the training device 520 without the function of executing instructions, which is not limited here.

[0114] In one possible implementation, the trained neural network (e.g., the updated pre-trained model in the embodiments of this application) can be deployed on a server, which can provide services to the client side through an application programming interface (API).

[0115] In this process, the terminal device can send relevant parameters (such as prompt information, including information about the target object, the environment in which the target object is located, and task information) to the server through the API provided by the cloud. The server can obtain the processing result (such as the control information of the target object) based on the received parameters and return the processing result to the terminal.

[0116] The description of the terminal and server can be found in the above embodiments, and will not be repeated here.

[0117] like Figure 3B This illustrates the process of using a cloud service provided by a cloud platform.

[0118] 1. Activate and purchase content moderation services.

[0119] 2. Users can download the software development kit (SDK) corresponding to the content moderation service. Cloud platforms usually provide multiple development versions of the SDK for users to choose from according to their development environment needs, such as JAVA version SDK, Python version SDK, PHP version SDK, Android version SDK, etc.

[0120] 3. After downloading the corresponding version of the SDK to their local machine according to their needs, users can import the SDK project into their local development environment, configure and debug it in the local development environment, and develop other functions in the local development environment to form an application that integrates the implementation of the target task (such as autonomous driving or robot posture control).

[0121] 4. During the application's use, it can trigger API calls related to the target task. When the application triggers the function of the target task, it initiates an API request to the running instance of the service related to the target task in the cloud environment. The API request carries prompt information, which is processed by the running instance in the cloud environment to obtain the processing result.

[0122] 5. The cloud environment returns the processing result to the application, thereby completing one method call provided in this embodiment of the application.

[0123] In addition, refer to Figure 3CThis application can also be applied to online learning scenarios deployed on target objects (such as robots or vehicles). Taking robots as an example, in online learning scenarios for robot skills, such as home scenarios, users can verbally assign a task to the robot face-to-face. The robot will make a few interactive attempts with the environment within a certain safe range (optionally, humans can intervene to provide feedback on the quality of its behavior so that the robot can adjust its behavior). The learned skill package is closer to the user's usage habits.

[0124] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0125] (1) Neural Network

[0126] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0127]

[0128] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0129] (2) Deep Neural Networks

[0130] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0131] (3) Reinforcement learning (RL), also known as reinforcement learning, evaluation learning or reinforcement learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how an agent learns strategies to maximize rewards or achieve specific goals in the process of interacting with the environment.

[0132] The common model for reinforcement learning is the standard Markov decision process (MDP). Depending on the conditions, reinforcement learning can be categorized into model-based reinforcement learning (RL) and model-free reinforcement learning (RL), as well as active reinforcement learning (RL) and passive reinforcement learning (RL). Variations of reinforcement learning include inverse reinforcement learning, hierarchical reinforcement learning, and reinforcement learning for partially observable systems. Algorithms used to solve reinforcement learning problems can be divided into policy search algorithms and value function algorithms. Deep learning models can be used in reinforcement learning, forming deep reinforcement learning.

[0133] (4) Loss Function

[0134] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0135] (5) Backpropagation algorithm

[0136] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0137] (6) Intelligent Agent

[0138] An intelligent agent is a concept in the field of artificial intelligence. Any entity capable of independent thought and interaction with its environment can be abstracted as an intelligent agent. The basic characteristics of an intelligent agent are: it can react to changes in its environment and automatically adjust its behavior and state; different intelligent agents can also interact with other intelligent agents according to their own intentions.

[0139] Object manipulation technologies based on natural language processing models (such as robot posture control or autonomous driving of vehicles) can be widely applied in scenarios such as robot posture control or autonomous driving of vehicles. Taking robots as an example, robot task skill learning is a challenging task. Existing robot skill learning in the field of natural language manipulation often requires the collection of large amounts of natural language data and the introduction of a large amount of human-in-the-loop feedback information, resulting in low efficiency, high cost, and slow learning. Therefore, it is hoped that the powerful reasoning ability of the basic model can be used to quickly learn the robot's complex decision-making skills.

[0140] In existing technologies, a pre-trained natural language base model is used to define a task through a prompt. After receiving prompt information such as task constraints, the base model outputs the corresponding decision behavior (or control information, such as instructions or scripts). When the model's output behavior is incorrect, humans use natural language feedback to allow the base model to correct the decision output, iterating repeatedly until the decision output is completely correct.

[0141] However, as the complexity of the tasks to be accomplished increases, existing technologies suffer from high skill learning costs, long learning cycles, and difficulty in learning complex skills.

[0142] To solve the above problems, refer to Figure 4 , Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application, such as... Figure 4 As shown in the embodiment of this application, a data processing method includes:

[0143] 401. Obtain the first prompt information; the first prompt includes information about the first task and information about the target object, wherein the first task is a task specified to be performed by the target object.

[0144] In this step, the execution subject of step 401 can be a training device (for example, the training device can be a terminal device or a server), as described in the above embodiments, and will not be repeated here.

[0145] In one possible implementation, some or all of the first prompt information may be user-inputted or detected by the interface based on the target object.

[0146] In one possible implementation, the first task can be defined using a prompt. That is, the first prompt information can include information about the first task. For example, the information about the first task can describe the execution effect of the first task, such as grabbing an object, organizing a certain area, or avoiding obstacles.

[0147] In one possible implementation, the information for the first task can be a textual description, that is, a description of the first task in the form of text, such as "grab the vase placed on the table".

[0148] In one possible implementation, the information for the first task can be image information, that is, the first task can be described in the form of an image. For example, the image content can be "a robotic arm grabs a vase placed on a table" or "a hand grabs a vase placed on a table".

[0149] In one possible implementation, the prompt information may also include information about the target object. For example, the target object information may include: the current posture of the target object (e.g., the state of each joint on the robot (position, angle, velocity, acceleration, etc.)). The target object information may also include: interface information of the manipulation components on the target object, and / or interface information of the sensing components on the target object. Here, the interface is the interface between the target object and the external environment; the sensing components can be sensors deployed on the target object; and the manipulation components can be movable joints, etc. For example, the target object information may be the robot's perception and control interface and its functional description (basic interfaces such as switching grippers, robotic arm movement, AGV movement, etc.).

[0150] In one possible implementation, the prompt information may further include information regarding the reward method (which may be simply referred to as the reward function information in this embodiment). Optionally, the reward function information may indicate the numerical range of the reward function (e.g., within the range of -5 to 5). Optionally, the reward function information may indicate the relationship between the execution result and the value of the reward function (e.g., a larger reward function indicates a better execution result).

[0151] In one possible implementation, the prompt information may also include information about the environment in which the target object is located, such as obstacle information in the robot's environment.

[0152] 402. Based on the information of the first task and the information of the target object, first control information is obtained through a pre-trained model; the first control information is used as control information for the target object when performing the first task.

[0153] In one possible implementation, the pre-trained model can be chatGPT, GPT4, or chatGLM. Furthermore, the pre-trained model can also be other types of large models, which are not limited in this application.

[0154] In one possible implementation, the pre-trained model can determine the control information based on prompt information such as the information of the input target object, the information of the first task, and the information of the environment in which the target object is located. Based on the determined control information, the model can manipulate the target object to perform the task.

[0155] Taking a robot as the target object as an example, the control information can be the rotation angle of the robot's joints, etc.

[0156] 403. Based on the first execution result obtained by the target object performing the first task based on the first control information, update the pre-trained model to obtain the updated pre-trained model.

[0157] In one possible implementation, a first reward value can be obtained based on the first execution result obtained by the target object performing the first task based on the first control information, and the pre-trained model can be updated based on the first reward value. The first reward value can be used to represent the performance quality of the first execution result compared to the first task.

[0158] For example, the first execution result can represent the state information of the target object after it has been manipulated through control information.

[0159] The following is a diagram illustrating how the reward value is determined:

[0160] In one possible implementation, a first reward value can be determined based on a first similarity between the first execution result and the information of the first task.

[0161] In one possible implementation, the first execution result may be, but is not limited to, an image of the environment in which the target object is located after the target object is manipulated by control information (e.g., an image of a physical or simulated environment, which may also be referred to as an execution result image in this application). The image may include the target object (all or key parts related to the first task, such as a robotic arm) and interactive objects of the target object indicated in the first task. For example, if the first task is to grab a vase on a table, the image may include, but is not limited to, the robotic arm, the table, and the vase on the table.

[0162] In one possible implementation, the reward value can be determined by the similarity between the information of the first execution result and the information of the first task. For example, the similarity between the information of the first execution result and the information of the first task can be determined by a preset distance calculation method.

[0163] In one possible implementation, the information of the first task includes textual description information related to the first task, and the first similarity is the similarity between the first execution result and the textual description information related to the first task.

[0164] In this scenario, users can define a task by inputting only text information without corresponding image information. In this case, the reward value for each step can be calculated using the cosine similarity between the task sentence's text embedding and the currently observed image embedding. This means that the closer the information within the current field of view is to the task's text description, the higher the reward value. The CLIP pre-trained model is used to calculate the task sentence's text embedding and the current image embedding. Both are one-dimensional vectors of length n. Optionally, the reward function `reward_text` is defined as follows:

[0165]

[0166] Where text_embedding is the task text vector, and image_embedding is the image vector of the currently observed image.

[0167] In one possible implementation, the information of the first task includes an execution result image related to the first task; then the first similarity can be the similarity between the first execution result and the execution result image related to the first task.

[0168] In this scenario, when defining a task, the user can simply input an image of the target state without any corresponding textual description. In this case, the reward value for each step can be the cosine similarity between the image vector of the target state (goal_image_embedding) and the vector of the currently observed image (image_embedding). This means that the closer the information within the current field of view is to the image information of the task, the larger the reward value. The CLIP pre-trained model can be used to calculate the image vector of the target state (goal_image_embedding) and the image vector of the currently observed image (image_embedding). Both are one-dimensional vectors of length n. Optionally, the reward function reward_image can be defined as:

[0169]

[0170] Where goal_image_embedding is the image vector of the task target state, and image_embedding is the image vector of the currently observed image.

[0171] In one possible implementation, the information of the first task includes textual description information related to the first task and an execution result image related to the first task; then the first similarity can be determined based on the similarity between the first execution result and the textual description information related to the first task, and the similarity between the first execution result and the execution result image related to the first task.

[0172] In this scenario, when defining a task, the user can simultaneously input both a text description of the task and an image of the task's target state. In this case, the `reward_text` from scenario 1 and the `reward_image` from scenario 2 can be considered together and weighted. The reward function `reward` is then defined as follows:

[0173] reward=α·reward_text+(1-α)·reward_image;

[0174] Here, α is a weight between 0 and 1. The larger α is, the more emphasis is placed on the text description, and the smaller α is, the more emphasis is placed on the content indicated in the task target status image.

[0175] It should be understood that the models for extracting text and image vectors mentioned above include, but are not limited to, the CLIP pre-trained model.

[0176] In this way, the target can continuously adjust its decision-making behavior based on reward information and new image observation information, learning new skills in a self-supervised manner. Repeating the above steps (which can include different tasks between repetitions) allows for the learning of a series of basic skills.

[0177] The aforementioned self-supervised learning approach enables large multimodal models to quickly learn skills. By agreeing on tasks and reward methods with the pre-trained model, the pre-trained model can form a self-supervised decision-making skill learning framework with the environment, eliminating the need for human intervention in feedback. Since the pre-trained base model itself has strong reasoning capabilities, compared to traditional reinforcement learning algorithms, the self-supervised learning method proposed in this application requires only a small number of interactions to learn a sub-skill.

[0178] 404. Obtain second prompt information; the second prompt includes information about the second task; the execution complexity of the second task is higher than that of the first task; the second task is a task to be performed by the specified target object.

[0179] In one possible implementation, the first task is a subtask that needs to be executed when performing the second task. The first task has lower complexity than the second task; that is, the first task is a basic task, and the second task is a more advanced task. For example, the first task could be a grasping action, and the second task could be a task of tidying up a desktop; or the first task could be a turning task, and the second task could be a task of obstacle avoidance. Optionally, the advanced task can be implemented by executing a combination of multiple basic tasks.

[0180] In one possible implementation, the second prompt information may further include: instruction information, which instructs the target object to perform the second task through a combination of control information from multiple learned tasks, including the first task. That is, when defining a task, a prompt can be used to inform the pre-trained model that, during the learning process, it can continuously and autonomously combine existing skills to form new skills. Specifically, when learning more complex skills, new and more complex skills will evolve based on previously learned skills. This approach improves the model's learning efficiency, and by starting with simple sub-skills and gradually evolving the learning from simple to complex task skills, the model can learn more complex skills.

[0181] 405. Based on the information of the second task and the information of the target object, second control information is obtained through the updated pre-trained model; the second control information is used as control information for the target object when performing the second task.

[0182] In one possible implementation, a second reward value can be obtained based on the second execution result obtained by the target object performing the second task based on the second control information and the information of the reward function, and the updated pre-trained model can be updated based on the second reward value.

[0183] In one possible implementation, the second control information can be obtained from the updated pre-trained model based on the information of the first task, the information of the target object, and the information of the environment.

[0184] 406. Update the updated pre-trained model based on the second execution result obtained by the target object performing the second task based on the second control information.

[0185] In one possible implementation, a second reward value can be determined based on a second similarity between the second execution result obtained by the target object performing the second task based on the second control information and the information of the second task, and the updated pre-trained model can be updated based on the second reward value.

[0186] In one possible implementation, the information of the second task includes textual description information related to the second task and an execution result image related to the second task; the second similarity is determined based on the similarity between the second execution result and the textual description information related to the second task, and the similarity between the second execution result and the execution result image related to the second task.

[0187] It should be understood that after the pre-trained model learns the skills related to the second task, a more complex task than the second task can be specified through a prompt, allowing the pre-trained model to learn more complex task execution capabilities based on skill combinations from multiple tasks, including the first and second tasks. For example, refer to... Figure 5 , Figure 5 This is a diagram illustrating a self-directed learning process for a skill.

[0188] In addition to being directly applicable to robot decision-making skills learning scenarios, the embodiments of this application can also be applied to other non-robot fields, such as learning skills in the Internet field (e.g., solving the problem of automatic page traversal in the testing field, low-code application development, etc.). The embodiments of this application can be applied to any skill learning involving permutations and combinations.

[0189] Reference Figure 6 , Figure 6 This is a flowchart illustrating a data processing method described from the perspective of model reasoning, such as... Figure 6 As shown, the method includes:

[0190] 601. Obtain target prompt information; the target prompt includes information about the target task.

[0191] 602. Based on the information of the target task, through methods such as... Figure 4 The updated pre-trained model obtained by the introduced data processing method yields target control information; the target control information is used as control information for the target object when performing the target task.

[0192] about Figure 6 The description of the corresponding embodiments can be found in [reference]. Figure 4 The process of model feedforward in the embodiments described herein is similar and will not be repeated here.

[0193] Reference Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application, such as... Figure 7 As shown, the device 700 includes:

[0194] The acquisition module 701 is used to acquire first prompt information; the first prompt includes information about a first task and information about a target object, wherein the first task is a task specified to be performed by the target object.

[0195] The specific description of the acquisition module 701 can be found in the descriptions of steps 401 and 404 in the above embodiments, and will not be repeated here.

[0196] The processing module 702 is configured to obtain first control information based on the information of the first task and the information of the target object through a pre-trained model; the first control information is used as control information for the target object when performing the first task.

[0197] Based on the first execution result obtained by the target object performing the first task based on the first control information, the pre-trained model is updated to obtain the updated pre-trained model;

[0198] The acquisition module is further configured to acquire second prompt information; the second prompt includes information about a second task; the execution complexity of the second task is higher than that of the first task; the second task is a task specified for the target object to be executed.

[0199] The processing module is further configured to obtain second control information based on the information of the second task and the information of the target object through the updated pre-trained model; the second control information is used as control information for the target object when performing the second task.

[0200] The updated pre-trained model is updated based on the second execution result obtained by the target object performing the second task based on the second control information.

[0201] The specific description of the processing module 702 can be found in the descriptions of steps 402, 403, 405 and 406 in the above embodiments, and will not be repeated here.

[0202] In one possible implementation, the first task is a subtask that needs to be executed when the second task is performed.

[0203] In one possible implementation, the first prompt information or the second prompt information further includes: information about the reward function; the information about the reward function indicates the numerical range of the reward function and / or the relationship between the execution result and the numerical value of the reward function;

[0204] The processing module is specifically configured to obtain a first reward value based on the first execution result obtained by the target object performing the first task based on the first control information and the information of the reward function, and update the pre-trained model based on the first reward value; or,

[0205] The processing module is specifically used for:

[0206] Based on the second execution result obtained by the target object performing the second task based on the second control information and the information of the reward function, a second reward value is obtained, and the updated pre-trained model is updated based on the second reward value.

[0207] In one possible implementation, the second prompt message also includes:

[0208] The instruction information instructs the target object to perform the second task by a combination of control information for multiple learned tasks, including the first task.

[0209] In one possible implementation, the first prompt or the second prompt may further include: information about the environment in which the target object is located;

[0210] The processing module is specifically used to: obtain first control information based on the information of the first task, the information of the target object, and the information of the environment through a pre-trained model; or,

[0211] The processing module is specifically used to: obtain second control information based on the information of the first task, the information of the target object, and the information of the environment through the updated pre-trained model.

[0212] In one possible implementation, the processing module is specifically configured to: determine a first reward value based on a first similarity between the first execution result obtained by the target object performing the first task based on the first control information and the information of the first task; and update the pre-trained model based on the first reward value to obtain an updated pre-trained model; or...

[0213] The processing module is specifically used to: determine a second reward value based on the second similarity between the second execution result obtained by the target object performing the second task based on the second control information and the information of the second task, and update the updated pre-trained model based on the second reward value.

[0214] In one possible implementation, the information of the first task includes textual description information related to the first task and an execution result image related to the first task; the first similarity is determined based on the similarity between the first execution result and the textual description information related to the first task, and the similarity between the first execution result and the execution result image related to the first task; or,

[0215] The information of the second task includes textual description information related to the second task and execution result images related to the second task; the second similarity is determined based on the similarity between the second execution result and the textual description information related to the second task, and the similarity between the second execution result and the execution result images related to the second task.

[0216] In one possible implementation, the information of the target object includes: interface information of the manipulation components on the target object, and / or interface information of the sensing components on the target object.

[0217] In one possible implementation, the pre-trained model is chatGPT, GPT4, or chatGLM.

[0218] In one possible implementation, the target object is a robot; the first task and the second task are the robot's posture control; or,

[0219] The target object is a vehicle; the first task and the second task are the autonomous driving of the vehicle.

[0220] Furthermore, embodiments of this application also provide a data processing apparatus, the apparatus comprising:

[0221] The acquisition module is used to acquire target prompt information; the target prompt includes information about the target task.

[0222] The processing module is used to, based on the information of the target task, perform operations such as... Figure 4 The updated pre-trained model obtained by the introduced data processing method yields target control information; the target control information is used as control information for the target object when performing the target task.

[0223] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 8 , Figure 8This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 800 can specifically be a mobile phone, tablet, laptop, smart wearable device, etc., and is not limited thereto. Specifically, the execution device 800 includes: a receiver 801, a transmitter 802, a processor 803, and a memory 804 (wherein the execution device 800 may have one or more processors 803). Figure 8 (Taking a processor as an example), the processor 803 may include an application processor 8031 ​​and a communication processor 8032. In some embodiments of this application, the receiver 801, transmitter 802, processor 803, and memory 804 may be connected via a bus or other means.

[0224] Memory 804 may include read-only memory and random access memory, and provides instructions and data to processor 803. A portion of memory 804 may also include non-volatile random access memory (NVRAM). Memory 804 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0225] The processor 803 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0226] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 803. Processor 803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 803 or by instructions in software form. Processor 803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 803 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 804, and processor 803 reads the information from memory 804 and, in conjunction with its hardware, completes the steps of the above method.

[0227] Receiver 801 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 802 can be used to output digital or character information; transmitter 802 can also be used to send instructions to the disk group to modify the data in the disk group.

[0228] In one embodiment of this application, the processor 803 is configured to execute... Figure 4 or Figure 6 The steps of the data processing method in the corresponding embodiment.

[0229] This application also provides a server; please refer to [link / reference]. Figure 9 , Figure 9This is a schematic diagram of a server structure provided in an embodiment of this application. Specifically, server 900 is implemented by one or more servers. Server 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 990 (e.g., one or more processors) and memory 932, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 942 or data 944. The memory 932 and storage media 930 can be temporary or persistent storage. The program stored in storage media 930 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 990 may be configured to communicate with storage media 930 and execute the series of instruction operations in storage media 930 on server 900.

[0230] Server 900 may also include one or more power supplies 920, one or more wired or wireless network interfaces 950, one or more input / output interfaces 958; or one or more operating systems 941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0231] In this embodiment, the central processing unit 990 is used to execute... Figure 4 or Figure 6 The steps of the data processing method in the corresponding embodiment.

[0232] This application also provides a computer program product including computer-readable instructions, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0233] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0234] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the steps related to model training in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0235] For details, please refer to Figure 10 , Figure 10 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1000. The NPU 1000 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1003, which is controlled by the controller 1004 to extract matrix data from the memory and perform multiplication operations.

[0236] In some implementations, the arithmetic circuit 1003 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1003 is a two-dimensional pulsating array. The arithmetic circuit 1003 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1003 is a general-purpose matrix processor.

[0237] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1008.

[0238] Unified memory 1006 is used to store input and output data. Weight data is directly transferred to weight memory 1002 via Direct Memory Access Controller (DMAC) 1005. Input data is also transferred to unified memory 1006 via DMAC.

[0239] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1009.

[0240] The Bus Interface Unit (BIU) 1010 is used by the instruction fetch memory 1009 to fetch instructions from external memory, and also by the memory access controller 1005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0241] The DMAC is mainly used to move input data from external memory DDR to unified memory 1006, or to weight data to weight memory 1002, or to input data to input memory 1001.

[0242] The vector computation unit 1007 includes multiple arithmetic processing units that, when needed, further process the output of the computation circuit, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0243] In some implementations, the vector computation unit 1007 can store the processed output vector in the unified memory 1006. For example, the vector computation unit 1007 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1003, such as linear interpolation of feature planes extracted by a convolutional layer, or, for example, a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 1007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1003, for example, for use in subsequent layers of the neural network.

[0244] The instruction fetch buffer 1009 connected to the controller 1004 is used to store the instructions used by the controller 1004;

[0245] Unified memory 1006, input memory 1001, weight memory 1002, and instruction fetch memory 1009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0246] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0247] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0248] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0249] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0250] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, The method includes: Obtain first prompt information; the first prompt information includes information about a first task and information about a target object, wherein the first task is a task specified to be performed by the target object; Based on the information of the first task and the information of the target object, first control information is obtained through a pre-trained model; the first control information is used as the control information for the target object when performing the first task. Based on the first execution result obtained by the target object performing the first task based on the first control information, the pre-trained model is updated to obtain the updated pre-trained model; Obtain second prompt information; the second prompt information includes information about a second task; the execution complexity of the second task is higher than that of the first task; the second task is a task to be performed on the specified target object; Based on the information of the second task and the information of the target object, second control information is obtained through the updated pre-trained model; the second control information is used as the control information for the target object when performing the second task. The updated pre-trained model is updated based on the second execution result obtained by the target object performing the second task based on the second control information.

2. The method according to claim 1, characterized in that, The first task is a subtask that needs to be executed when the second task is performed.

3. The method according to claim 1 or 2, characterized in that, The first prompt information or the second prompt information further includes: information about the reward function; the information about the reward function indicates the numerical range of the reward function and / or the relationship between the execution result and the numerical value of the reward function; The step of updating the pre-trained model based on the first execution result obtained by the target object performing the first task based on the first control information includes: obtaining a first reward value based on the first execution result obtained by the target object performing the first task based on the first control information and the information of the reward function, and updating the pre-trained model based on the first reward value; or, The step of updating the updated pre-trained model based on the second execution result obtained by the target object performing the second task based on the second control information includes: Based on the second execution result obtained by the target object performing the second task based on the second control information and the information of the reward function, a second reward value is obtained, and the updated pre-trained model is updated based on the second reward value.

4. The method according to any one of claims 1 to 2, characterized in that, The second prompt information also includes: The instruction information instructs the target object to perform the second task by a combination of control information for multiple learned tasks, including the first task.

5. The method according to any one of claims 1 to 2, characterized in that, The step of updating the pre-trained model based on the first execution result obtained by the target object performing the first task based on the first control information, to obtain an updated pre-trained model, includes: determining a first reward value based on a first similarity between the first execution result obtained by the target object performing the first task based on the first control information and the information of the first task; and updating the pre-trained model based on the first reward value to obtain an updated pre-trained model; or... The step of updating the updated pre-trained model based on the second execution result obtained by the target object performing the second task based on the second control information includes: determining a second reward value based on the second similarity between the second execution result obtained by the target object performing the second task based on the second control information and the information of the second task, and updating the updated pre-trained model based on the second reward value.

6. The method according to claim 5, characterized in that, The information of the first task includes textual description information related to the first task and an execution result image related to the first task; the first similarity is determined based on the similarity between the first execution result and the textual description information related to the first task, and the similarity between the first execution result and the execution result image related to the first task; or, The information for the second task includes textual descriptions related to the second task and images showing the execution results related to the second task; The second similarity is determined based on the similarity between the second execution result and the textual description information related to the second task, and the similarity between the second execution result and the execution result image related to the second task.

7. The method according to any one of claims 1 to 2, characterized in that, The information of the target object includes: the interface information of the control components on the target object, and / or the interface information of the sensing components on the target object.

8. The method according to any one of claims 1 to 2, characterized in that, The pre-trained model is chatGPT, GPT4, or chatGLM.

9. The method according to any one of claims 1 to 2, characterized in that, The target object is a robot; the first task and the second task are the robot's posture control; or, The target object is a vehicle; the first task and the second task are the autonomous driving of the vehicle.

10. A data processing method, characterized in that, The method includes: Obtain target prompt information; the target prompt information includes information about the target task; Based on the information of the target task, target control information is obtained through the updated pre-trained model obtained by any one of the methods in claims 1 to 9; the target control information is used as control information for the target object when performing the target task.

11. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire first prompt information; the first prompt information includes information about a first task and information about a target object, wherein the first task is a task specified to be performed by the target object. The processing module is configured to obtain first control information based on the information of the first task and the information of the target object through a pre-trained model; the first control information is used as control information for the target object when performing the first task. Based on the first execution result obtained by the target object performing the first task based on the first control information, the pre-trained model is updated to obtain the updated pre-trained model; The acquisition module is further configured to acquire second prompt information; the second prompt information includes information about a second task; the execution complexity of the second task is higher than that of the first task; the second task is a task specified to be performed by the target object; The processing module is further configured to obtain second control information based on the information of the second task and the information of the target object through the updated pre-trained model; the second control information is used as control information for the target object when performing the second task. The updated pre-trained model is updated based on the second execution result obtained by the target object performing the second task based on the second control information.

12. The apparatus according to claim 11, characterized in that, The first task is a subtask that needs to be executed when the second task is performed.

13. The apparatus according to claim 11 or 12, characterized in that, The first prompt information or the second prompt information further includes: information about the reward function; the information about the reward function indicates the numerical range of the reward function and / or the relationship between the execution result and the numerical value of the reward function; The processing module is specifically configured to obtain a first reward value based on the first execution result obtained by the target object performing the first task based on the first control information and the information of the reward function, and update the pre-trained model based on the first reward value; or, The processing module is specifically used for: Based on the second execution result obtained by the target object performing the second task based on the second control information and the information of the reward function, a second reward value is obtained, and the updated pre-trained model is updated based on the second reward value.

14. The apparatus according to any one of claims 11 to 12, characterized in that, The second prompt information also includes: The instruction information instructs the target object to perform the second task by a combination of control information for multiple learned tasks, including the first task.

15. The apparatus according to any one of claims 11 to 12, characterized in that, The processing module is specifically configured to: determine a first reward value based on a first similarity between the first execution result obtained by the target object performing the first task based on the first control information and the information of the first task; and update the pre-trained model based on the first reward value to obtain an updated pre-trained model; or... The processing module is specifically used to: determine a second reward value based on the second similarity between the second execution result obtained by the target object performing the second task based on the second control information and the information of the second task, and update the updated pre-trained model based on the second reward value.

16. The apparatus according to claim 15, characterized in that, The information of the first task includes textual description information related to the first task and an execution result image related to the first task; the first similarity is determined based on the similarity between the first execution result and the textual description information related to the first task, and the similarity between the first execution result and the execution result image related to the first task; or, The information for the second task includes textual descriptions related to the second task and images showing the execution results related to the second task; The second similarity is determined based on the similarity between the second execution result and the textual description information related to the second task, and the similarity between the second execution result and the execution result image related to the second task.

17. The apparatus according to any one of claims 11 to 12, characterized in that, The information of the target object includes: the interface information of the control components on the target object, and / or the interface information of the sensing components on the target object.

18. The apparatus according to any one of claims 11 to 12, characterized in that, The pre-trained model is chatGPT, GPT4, or chatGLM.

19. The apparatus according to any one of claims 11 to 12, characterized in that, The target object is a robot; the first task and the second task are the robot's posture control; or, The target object is a vehicle; the first task and the second task are the autonomous driving of the vehicle.

20. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire target prompt information; the target prompt information includes information about the target task. The processing module is configured to obtain target control information based on the information of the target task, using an updated pre-trained model obtained by any one of the methods described in claims 1 to 9; the target control information is used as control information for the target object when performing the target task.

21. A data processing apparatus, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to retrieve the code and perform the method as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method according to any one of claims 1 to 10.

23. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 10.

Images