Data processing method and device, intelligent agent training method and device and electronic equipment

Abstract

The invention discloses a data processing method and device, an agent training method and device and electronic equipment. The method comprises the following steps: acquiring environment data and a working instruction; a multifunctional intelligent agent is adopted, a target model combination corresponding to the environment data and the work instruction is determined, the multifunctional intelligent agent comprises a plurality of machine learning models with different functions, and the target model combination is used for representing a combination mode and input and output information between the machine learning models; the multifunctional agent is obtained by training in an inverse reinforcement learning mode according to the historical environment data after generalization operation and expert demonstration information corresponding to the historical environment data, and the expert demonstration information comprises a target model combination which is set artificially and corresponds to the historical environment data; and determining an operation instruction corresponding to the work instruction by adopting the target model combination according to the environment data. The technical problem that an intelligent agent is difficult to deal with complex and changeable environments and tasks in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a data processing method and an intelligent agent training method, device and electronic device. Background Art

[0002] With the continuous advancement of technology and the continuous expansion of application fields, the functions and performance of multifunctional intelligent agents are constantly improving, bringing more convenience and benefits to human life and work. However, the relevant technology currently lacks a method for conveniently converting the work skills of multiple agents for different complex work scenarios, that is, the intelligent agents in the relevant technology have technical problems that are difficult to cope with complex and changing environments and tasks.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and an intelligent agent training method, device and electronic device to at least solve the technical problem in related technologies that intelligent agents have difficulty in coping with complex and changing environments and tasks.

[0005] According to one aspect of an embodiment of the present application, a data processing method is provided, including: obtaining environmental data and work instructions, wherein the environmental data is used to characterize information about the environment in which a multifunctional intelligent agent is located; using the multifunctional intelligent agent to determine a target model combination corresponding to the environmental data and the work instructions, wherein the multifunctional intelligent agent includes multiple machine learning models with different functions, and the target model combination is used to characterize the combination mode and input-output information between the machine learning models, and the multifunctional intelligent agent is trained by inverse reinforcement learning based on historical environmental data after generalization operations and expert demonstration information corresponding to the historical environmental data, and the expert demonstration information includes: an artificially set target model combination corresponding to the historical environmental data; using the target model combination to determine, based on the environmental data, an operation instruction corresponding to the work instruction, wherein the operation instruction is used to control the multifunctional intelligent agent to perform operations that can meet the task requirements corresponding to the work instruction.

[0006] Optionally, the training steps of the multifunctional intelligent agent include: obtaining historical environmental data in the environment corresponding to the environmental data, and expert demonstration information corresponding to the historical environmental data; using transfer learning to generalize the historical environmental data and expert demonstration information, wherein the generalization operation is used to extract common features from different historical environmental data or expert demonstration information; using inverse reinforcement learning and reinforcement learning to train the multifunctional intelligent agent based on the historical environmental data and expert demonstration information after generalization.

[0007] Optionally, the multifunctional intelligent agent is trained based on the historical environmental data and expert demonstration information after generalization operations by inverse reinforcement learning and reinforcement learning, including: performing inverse reinforcement learning on the historical environmental data and expert demonstration information after generalization operations to obtain a reward function, wherein the reward function is used to characterize the reward value corresponding to the multifunctional intelligent agent performing different predicted actions under different environmental data, and the predicted action is used to determine the target model combination; using the multifunctional intelligent agent to predict the target model combination corresponding to the historical environmental data based on the reward function; determining the error loss between the predicted target model combination and the target model combination corresponding to the expert demonstration information; updating the reward function based on the error loss, and using the multifunctional intelligent agent to repeat the above prediction process based on the updated reward function until the error loss is less than a preset error threshold.

[0008] Optionally, obtaining environmental data and work instructions includes: obtaining multimodal data collected by sensor equipment of the multifunctional intelligent body, wherein the sensor equipment includes at least one of the following: a visual sensor, an audio sensor, and a balancer sensor; determining a data type of each data in the multimodal data, converting the data into a preset standard format corresponding to the data type, and integrating the multimodal data converted into the preset standard format into a data file in a preset format; performing a generalization operation on the data in the data file to obtain environmental data, wherein the generalization operation is used to extract common features from different data.

[0009] Optionally, when the historical environment data or the data file contains image data, the generalization operation includes: using a diffusion model to extract feature values ​​of the image data and determine feature texts corresponding to the feature values, wherein the diffusion model includes at least one of the following: a Stable Diffusion model; determining the part of speech of each word in the feature text, wherein the part of speech includes: noun; determining the common noun combination between the feature texts corresponding to different image data, and obtaining the features of the image data after the generalization operation.

[0010] Optionally, the method also includes: determining an operation mode of the multifunctional intelligent agent, wherein the operation mode includes: a working mode and a learning mode, wherein, when the operation mode is the working mode, the multifunctional intelligent agent is used to respond to work instructions and perform operations that can meet the task requirements corresponding to the work instructions, and when the operation mode is the learning mode, the multifunctional intelligent agent is used to perform autonomous learning based on the multimodal data collected by the sensor device.

[0011] Optionally, the multifunctional intelligent body also includes: an operation module, the operation module is used to receive operation instructions and execute operation actions corresponding to the operation instructions, wherein the operation action includes at least one of the following: a robot arm movement action, an audio playback action.

[0012] According to another aspect of an embodiment of the present application, there is also provided an agent training method, comprising: obtaining environmental data and expert demonstration information corresponding to the environmental data, wherein the environmental data is used to characterize information about the environment in which the multifunctional agent is located, the multifunctional agent comprises a plurality of machine learning models with different functions, and the expert demonstration information comprises: an artificially set target model combination corresponding to the environmental data, the target model combination is used to characterize the combination mode and input and output information between machine learning models; performing generalization operations on the environmental data and the expert demonstration information by means of transfer learning, wherein the generalization operation is used to extract common features from different environmental data or expert demonstration information; and training the multifunctional agent by means of inverse reinforcement learning and reinforcement learning based on the environmental data and the expert demonstration information after the generalization operation to obtain a target multifunctional agent, wherein the target multifunctional agent is used to determine the target model combination corresponding to the environmental data.

[0013] According to another aspect of the embodiment of the present application, a data processing device is also provided, including: a data acquisition module, used to acquire environmental data and work instructions, wherein the environmental data is used to characterize information about the environment in which the multifunctional intelligent agent is located; a data analysis module, used to use the multifunctional intelligent agent to determine a target model combination corresponding to the environmental data and the work instructions, wherein the multifunctional intelligent agent includes multiple machine learning models with different functions, and the target model combination is used to characterize the combination mode and input and output information between the machine learning models, and the multifunctional intelligent agent is trained by inverse reinforcement learning based on historical environmental data after generalization operations and expert demonstration information corresponding to the historical environmental data, and the expert demonstration information includes: an artificially set target model combination corresponding to the historical environmental data; an instruction execution module, used to use the target model combination to determine the operation instructions corresponding to the work instructions based on the environmental data, wherein the operation instructions are used to control the multifunctional intelligent agent to perform operations that can meet the task requirements corresponding to the work instructions.

[0014] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the program executes a data processing method or an intelligent agent training method when it is run.

[0015] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes a data processing method or an intelligent agent training method by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, which implements the steps of the data processing method or the intelligent agent training method when the computer program is executed by a processor.

[0017] In an embodiment of the present application, environmental data and work instructions are obtained, wherein the environmental data is used to characterize information about the environment in which the multifunctional intelligent agent is located; a target model combination corresponding to the environmental data and the work instruction is determined by using the multifunctional intelligent agent, wherein the multifunctional intelligent agent includes multiple machine learning models with different functions, and the target model combination is used to characterize the combination mode and input and output information between the machine learning models. The multifunctional intelligent agent is trained by inverse reinforcement learning based on historical environmental data after generalization operations and expert demonstration information corresponding to the historical environmental data, and the expert demonstration information includes: a target model combination corresponding to the historical environmental data set manually; the target model combination is used to determine the operation instructions corresponding to the work instruction based on the environmental data, wherein the operation instructions are used to control the multifunctional intelligent agent to perform operations that can meet the task requirements corresponding to the work instruction. By combining technologies such as transfer learning and inverse reinforcement learning, the intelligent agent can automatically learn and transform skills of complex environments and instructions, thereby achieving the purpose of improving the generalization ability, flexibility and performance of the system, thereby solving the problem that the intelligent agent in the related technology is difficult to cope with complex and changeable environments and tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a method for data processing or agent training according to an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of a data processing method flow provided according to an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of a processing flow of a multifunctional intelligent agent capable of autonomous learning based on a model set according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a training method flow of a multifunctional intelligent agent provided according to an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of a method flow of information processing provided according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a method flow for training an intelligent agent provided according to an embodiment of the present application;

[0025] Figure 7 It is a structural diagram of a data processing device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] In order to facilitate those skilled in the art to better understand the embodiments of the present application, some technical terms or nouns involved in the embodiments of the present application are explained as follows:

[0029] Multifunctional agents: agents that have multiple functions and can complete multiple tasks. Such agents usually integrate multiple advanced technologies and components to achieve their diverse functions.

[0030] Stable Diffusion: is the most advanced model in the Diffusion diffusion model. It uses a more stable, controllable and efficient method to generate high-quality images. There are significant improvements in the quality, speed and cost of generated images, so this model can directly generate images on consumer-grade graphics cards, up to at least 512*512 pixels. At present, the application of Stable Diffusion is no longer limited to the field of image generation. It is also widely used in natural language processing, audio and video generation, and the technologies involved include:

[0031] Intelligent agents can be divided into many types according to their application fields and functional characteristics, such as industrial intelligent agents, service intelligent agents, special intelligent agents, etc. Multifunctional intelligent agents usually belong to the category of service intelligent agents or special intelligent agents, and have multiple functions and task execution capabilities.

[0032] Artificial intelligence is one of the core technologies of multifunctional intelligent agents. By integrating artificial intelligence algorithms and models, intelligent agents can understand and process complex information such as images, sounds, texts, etc., and improve their performance and behavior through learning and optimization.

[0033] Sensors are important tools for multifunctional intelligent agents to perceive the environment. By carrying various sensors, such as visual sensors, tactile sensors, force sensors, etc., intelligent agents can perceive changes in the surrounding environment and the properties of objects in real time. Sensor technology enables intelligent agents to accurately obtain environmental information and provide a reliable basis for subsequent decision-making and execution.

[0034] The control system is the brain of the multifunctional intelligent agent, responsible for processing sensor information, executing decisions and controlling the movement of the intelligent agent. By integrating advanced control algorithms and optimization algorithms, the control system can ensure that the intelligent agent performs tasks stably and accurately. The control system also has adaptive and learning capabilities, and can automatically adjust control strategies and optimize parameters according to environmental changes and task requirements.

[0035] In summary, the background technology of multifunctional intelligent agents involves multiple fields and multiple aspects. The continuous development and integration of these technologies provide strong support and guarantee for the development of multifunctional intelligent agents. With the continuous advancement of technology and the continuous expansion of application fields, the functions and performance of multifunctional intelligent agents will continue to improve, bringing more convenience and benefits to human life and work. However, the relevant technology currently lacks a method for conveniently converting the work skills of multiple agents for different complex work scenarios, that is, the intelligent agents in the relevant technology have technical problems that are difficult to cope with complex and changing environments and tasks.

[0036] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0037] According to an embodiment of the present application, an embodiment of a method for data processing is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1FIG. 1 shows a hardware structure block diagram of a computer terminal (or electronic device) for implementing a data processing method or an intelligent agent training method. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0039] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0043] In the above operating environment, the embodiment of the present application provides a data processing method. Figure 2 is a schematic diagram of a data processing method flow according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0044] Step S202, obtaining environmental data and work instructions, wherein the environmental data is used to represent information about the environment in which the multifunctional intelligent agent is located;

[0045] Step S204: using a multifunctional intelligent agent to determine a target model combination corresponding to the environmental data and the work instruction, wherein the multifunctional intelligent agent includes a plurality of machine learning models with different functions, and the target model combination is used to characterize the combination mode and input and output information between the machine learning models. The multifunctional intelligent agent is trained by inverse reinforcement learning based on the historical environmental data after generalization operation and the expert demonstration information corresponding to the historical environmental data, and the expert demonstration information includes: a target model combination corresponding to the historical environmental data set manually;

[0046] Step S206, using the target model combination, based on the environmental data, determine the operation instructions corresponding to the work instructions, wherein the operation instructions are used to control the multifunctional intelligent agent to perform operations that can meet the task requirements corresponding to the work instructions.

[0047] Through the above steps, by combining transfer learning, inverse reinforcement learning and other technologies, the intelligent agent can automatically learn and transfer skills of complex environments and instructions, thereby improving the generalization ability, flexibility and performance of the system, and thus solving the problem that the intelligent agents in related technologies are difficult to cope with complex and changeable environments and task technologies.

[0048] The data processing method in steps S202 to S206 of the embodiment of the present application is further introduced below.

[0049] Figure 3 is a schematic diagram of a processing flow of a multifunctional intelligent agent capable of autonomous learning based on a model set according to an embodiment of the present application, such as Figure 3 shown.

[0050] In an embodiment of the present application, the multifunctional intelligent agent system includes: sensor input devices such as visual encoders (sensors), audio encoders (sensors), balancer encoders (sensors); information identification, differentiation, and integration modules; intelligent agents; model sets such as third-person perspective imitation learning models, inverse reinforcement learning models, and large language models; and operation modules. Through this system, information can be input from the sensor input device, and the information processing module will perform identification, differentiation, and integration, and then pass it to the intelligent agent for analysis to generate corresponding model combinations, data input to the model, and data output by the model (i.e., target model combinations), and then pass it to the operation module to perform foot, hand, audio, and other actions. The above process is described in detail below.

[0051] In some embodiments of the present application, the method includes the following steps: determining an operating mode of the multifunctional intelligent agent, wherein the operating mode includes: a working mode and a learning mode, wherein, when the operating mode is the working mode, the multifunctional intelligent agent is used to respond to work instructions and perform operations that can meet the task requirements corresponding to the work instructions, and when the operating mode is the learning mode, the multifunctional intelligent agent is used to perform autonomous learning based on the multimodal data collected by the sensor device.

[0052] Specifically, in this embodiment, the multifunctional intelligent agent system includes two operating modes: working mode and learning mode. In the learning mode, the multifunctional intelligent agent can autonomously enter the learning mode and perform autonomous reinforcement learning through information collected by sensors. When receiving a work instruction, the multifunctional intelligent agent system will autonomously exit the learning mode and enter the working mode to work.

[0053] The following first introduces the training process of the intelligent agent in the learning mode. The specific steps are as follows.

[0054] In some embodiments of the present application, the training steps of the multifunctional intelligent agent include: obtaining historical environmental data in the environment corresponding to the environmental data, and expert demonstration information corresponding to the historical environmental data; using transfer learning to generalize the historical environmental data and expert demonstration information, wherein the generalization operation is used to extract common features from different historical environmental data or expert demonstration information; using inverse reinforcement learning and reinforcement learning to train the multifunctional intelligent agent based on the historical environmental data and expert demonstration information after generalization.

[0055] Specifically, the intelligent agent is a model body that combines reverse learning with transfer learning, such as Figure 4 As shown in the figure, the intelligent agent can encode the existing model set, the information processed by the information processing module and the information output by the model, distinguish the model composition, data input and output through encoding, and then pass the encoding as the expert demonstration information of the intelligent agent together with the environmental parameters to the reverse learning model of the intelligent agent for learning. The specific process is as follows.

[0056] First, obtain the historical environmental data in the environment corresponding to the environmental data, that is, use the characteristic value of the environmental image identified by the information processing module, and form the environmental parameters of learning or work. Then, obtain the corresponding expert demonstration information. In this embodiment, the expert demonstration information can use the optimal model combination designed manually and the corresponding coding of input and output data, for example: A001005003B002..... Among them, A001 is the coding of the first module, 005 is the coding corresponding to the information input of the first module, 003 is the coding of the output information of the first module operation, and B002 is the coding of the second module. The corresponding information is integrated into a JOSN format file.

[0057] Afterwards, the above environmental data and expert demonstration information can be generalized through transfer learning, as follows.

[0058] In some embodiments of the present application, performing a generalization operation includes the following steps: using a diffusion model to extract feature values ​​of image data and determine feature texts corresponding to the feature values, wherein the diffusion model includes at least one of the following: a StableDiffusion model; determining the part of speech of each word in the feature text, wherein the part of speech includes: noun; determining the common noun combination between the feature texts corresponding to different image data, and obtaining the features of the image data after the generalization operation.

[0059] Specifically, a diffusion model (for example, Stable Diffusion) can be used in the information processing module to extract the feature values ​​of the image, and then in the agent transfer learning, common feature values ​​can be extracted for a small number of similar images (for the feature value text of the extracted image, noun extraction is first performed, and then the common noun combination is obtained by comparison, and the final common feature value is obtained, and "learning" or "working" instructions are added) and the encoding of expert demonstration information combination, information input and information output is jointly generalized.

[0060] In addition, in this embodiment, a large language model (such as ChatGLM3-6b) can be used in the information processing module to perform freeze fine-tuning training on a small amount of instruction corpus with a loss of about 0.02. For questions with the same semantics as those in the instruction corpus, a unique corresponding instruction will be generated (such as generating a "learning" or "work" instruction).

[0061] After obtaining the generalized environmental data and expert demonstration information, inverse reinforcement learning can be performed to obtain the reward function, and then the generalized environmental data and reward function can be used for reinforcement learning. The specific steps are as follows.

[0062] In some embodiments of the present application, inverse reinforcement learning and reinforcement learning are used to train a multifunctional intelligent agent based on historical environmental data and expert demonstration information after generalization operations, including the following steps: inverse reinforcement learning is performed on the historical environmental data and expert demonstration information after generalization operations to obtain a reward function, wherein the reward function is used to characterize the reward value corresponding to the multifunctional intelligent agent performing different predicted actions under different environmental data, and the predicted action is used to determine the target model combination; using the multifunctional intelligent agent, based on the reward function, predicting the target model combination corresponding to the historical environmental data; determining the error loss between the predicted target model combination and the target model combination corresponding to the expert demonstration information; based on the error loss, updating the reward function, and using the multifunctional intelligent agent, repeating the above prediction process based on the updated reward function until the error loss is less than a preset error threshold.

[0063] Specifically, the generalized information (the common characteristic values ​​of environmental data, expert demonstration information as a model combination and a string of codes for information input and output) can be subjected to inverse reinforcement learning to obtain a reward function; the generalized environmental data and reward function are then subjected to reinforcement learning, and finally the optimal model combination and a string of codes for information input and output (that is, the above-mentioned target model combination) are obtained. After inverse reinforcement learning and reinforcement learning, the intelligent agent can automatically generate the optimal model combination and the encoding of input and output data for different environments and instructions, that is, obtain the optimal training model under this change.

[0064] After the model is trained for the environment, it can then, in working mode, perform skill conversion (optimal model combination and input and output data or optimal training model and optimal working model combination and input and output data) based on the environmental data and work instructions in the environment to adapt to learning or working in multiple instructions and multiple environments. This intelligent agent can solve problems such as the intelligent agent environment, single skills, and collaborative work between multiple agents, making this multifunctional intelligent agent more adaptable to learning and working in complex environments.

[0065] The following is an introduction to the processing flow of the intelligent agent in working mode.

[0066] First, the signal collected by the sensor is received by the sensor input device to generate information in a corresponding format (environmental data and work instructions). The specific steps are as follows.

[0067] In some embodiments of the present application, obtaining environmental data and work instructions includes the following steps: obtaining multimodal data collected by the sensor equipment of the multifunctional intelligent body, wherein the sensor equipment includes at least one of the following: a visual sensor, an audio sensor, and a balancer sensor; determining the data type of each data in the multimodal data, converting the data into a preset standard format corresponding to the data type, and integrating the multimodal data converted into the preset standard format into a data file in a preset format; performing a generalization operation on the data in the data file to obtain environmental data, wherein the generalization operation is used to extract common features from different data.

[0068] Specifically, Figure 5 As shown, the information processing module receives the information transmitted by the sensor input device (i.e., the multimodal data mentioned above), performs information processing such as identification and differentiation on the information, and then integrates the processed information into a data file in a preset format (e.g., a JOSN format file); then, the features transmitted by the sensor (i.e., the data in the data file) can be generalized through transfer learning, the number of learning and working environments can be increased, and then transmitted to the intelligent agent as environmental parameters.

[0069] Afterwards, the multifunctional intelligent agent that has undergone inverse reinforcement learning and reinforcement learning can determine the target model combination (the encoding of the optimal model combination and input and output data) corresponding to the environmental data and work instructions for the generalized environmental data. In this embodiment, the model combination, input and output data are encoded to facilitate the identification and control of the model and data. According to this string of codes, the model is called to form a model group, and then the information transmitted by the information processing module corresponding to the code is called to pass it to the corresponding model for processing and output.

[0070] After the intelligent agent has the optimal training model and the optimal working model combination and input and output data, it can work normally in the corresponding working environment and output operation instructions to the operation module, where the operation module is used to receive the operation instructions and execute the operation actions corresponding to the operation instructions, where the operation actions include at least one of the following: robotic arm movement action, audio playback action.

[0071] The embodiment of the present application uses an artificially designed optimal model combination and input and output data to first generalize the environmental information and expert demonstration through transfer learning, then perform inverse reinforcement learning on the generalized information to obtain a reward function, and then perform reinforcement learning on the generalized environmental data and reward function to finally obtain the optimal model combination and input data. After inverse reinforcement learning, the intelligent agent can automatically perform skill conversion on instructions and the environment (generating the optimal model combination and input data is a method of skill conversion), automatically learn the demonstration information of experts in this environment, and finally obtain the optimal actor (intelligent agent) to perform the work.

[0072] The above scheme is illustrated below by combining specific cases.

[0073] Taking the multifunctional intelligent body playing golf as an example, first, the sensor input device receives the signal collected by the sensor to generate information in the corresponding format; the information processing module receives the information from the sensor input device, identifies and distinguishes the information, and then integrates the processed information into a JOSN format file.

[0074] Before the intelligent agent runs, it must first be subjected to inverse reinforcement learning. The learning process is as follows: obtain the environmental image (environmental data) identified by the information processing module, and the corresponding encoding of the optimal model combination and input data designed manually, that is, the expert example information (for example: A001005.....); generalize the environmental information and expert demonstration through transfer learning (for example: extract common features and A001005.....); then perform inverse reinforcement learning on the generalized information (such as: extract common features "grass, water...", add them to the "learning" or "work" field, and combine them with the encoding A001005003.....) to obtain the reward function;

[0075] After obtaining the reward function, the generalized environmental data (such as extracting common features "grass, water..." and "learning" or "working") and the reward function are reinforced to learn, and finally the optimal model combination, input and output data (such as a string of codes: A001005003.....) are obtained, and the outputs between the models are combined into the data processed by the information processing module. After autonomous learning, the intelligent agent can automatically generate the optimal model combination and input and output data encoding for the (learning or working) environment; the expert demonstration information in this learning environment is automatically learned using the optimal learning model combination and input and output data, and finally the optimal training model is obtained; the intelligent agent has the optimal training model and the optimal working model combination and input and output data, and can work normally in the corresponding working environment and output instructions to the operation module; the operation module receives the instructions output by the optimal model combination and the information from the sensor of the information processing module, and performs corresponding operations to play golf.

[0076] This application scheme uses transfer learning to fine-tune the pre-trained model on the new task, which can effectively utilize existing knowledge and reduce the need for labeling new data, thereby speeding up the model training and improving the generalization ability on new tasks. At the same time, the expert's demonstration behavior is converted into a reward function through inverse reinforcement learning, which further enhances the model's ability to understand complex environments and instructions, allowing the model to simulate expert behavior more accurately. It enables the intelligent agent to automatically learn from expert demonstrations and convert these skills into the optimal model combination and input data. This process not only improves the autonomy and adaptability of the intelligent agent, but also significantly reduces the cost of manual intervention. The intelligent agent can quickly generate the optimal response strategy for different instructions and environments, thereby improving the flexibility and efficiency of the overall system. The combination of reinforcement learning and inverse reinforcement learning enables the model to comprehensively consider environmental information and expert experience in the decision-making process, so as to make more reasonable and efficient decisions. By continuously optimizing the reward function and reinforcement learning strategy, the model can continuously improve its performance and adapt to more complex and changing environments and tasks.

[0077] In addition, the use of Stable Diffusion to extract common features in the present application scheme can efficiently extract common features from a small number of similar images, and these features play a key role in understanding and describing the content of the image. In this way, the cost of data collection and processing can be greatly reduced, while improving the accuracy and efficiency of feature extraction. Moreover, the extracted common features are generalized with the "instructions" and "expert demonstration" encodings, so that the model can better understand complex instructions and demonstration behaviors. This generalization process enhances the semantic understanding ability of the model, enabling it to more accurately understand the user's intentions and needs.

[0078] The present application scheme encodes the data, model output and model processed by the information processing module so that all operation instructions can be identified and controlled under a unified framework. This unified operation instruction method reduces the complexity and maintenance cost of the system and improves the scalability and maintainability of the system. A series of operation instructions such as recognition, control and input implemented by coding enable the system to flexibly adapt to different application scenarios and task requirements. By adjusting the coding rules and parameter settings, the expansion and optimization of system functions can be easily achieved. The coding process can optimize data transmission and processing efficiency, reduce data redundancy and transmission delay. This is of great significance for improving the overall performance and response speed of the system, especially in scenarios where large amounts of data need to be processed in real time.

[0079] In summary, this application scheme combines advanced technologies such as transfer learning, inverse reinforcement learning and stable diffusion model to achieve automatic learning and skill conversion of intelligent agents for complex environments and instructions, while improving the generalization ability, flexibility and performance of the system. Multifunctional intelligent agents can improve production efficiency and automation level, cope with complex and changing environments and tasks, improve the quality of human life and service level, and promote scientific and technological progress and industrial development.

[0080] According to an embodiment of the present application, an embodiment of an intelligent agent training method is also provided. Figure 6 is a schematic diagram of a method flow for training an intelligent agent according to an embodiment of the present application. Figure 6 As shown, the method comprises the following steps:

[0081] Step S602, obtaining environmental data and expert demonstration information corresponding to the environmental data, wherein the environmental data is used to represent information about the environment in which the multifunctional intelligent agent is located, the multifunctional intelligent agent includes multiple machine learning models with different functions, and the expert demonstration information includes: a combination of target models corresponding to the environmental data set manually, the combination of target models is used to represent the combination mode and input and output information between the machine learning models;

[0082] Step S604, using transfer learning to perform a generalization operation on the environmental data and the expert demonstration information, wherein the generalization operation is used to extract common features from different environmental data or expert demonstration information;

[0083] Step S606, using inverse reinforcement learning and reinforcement learning, the multifunctional agent is trained based on the environmental data and expert demonstration information after generalization operations to obtain a target multifunctional agent, wherein the target multifunctional agent is used to determine the target model combination corresponding to the environmental data.

[0084] It should be noted that the agent training method provided in this embodiment is similar to Figure 2 The data processing method shown corresponds to a method embodiment, therefore, the relevant explanations and descriptions of the above data processing method are also applicable to the embodiments of the present application and will not be repeated here.

[0085] According to an embodiment of the present application, an embodiment of a data processing device is also provided. Figure 7 is a structural diagram of a data processing device provided according to an embodiment of the present application. Figure 7 As shown, the device comprises:

[0086] A data acquisition module 70, used to acquire environmental data and work instructions, wherein the environmental data is used to characterize information about the environment in which the multifunctional intelligent body is located;

[0087] The data analysis module 72 is used to use a multifunctional intelligent agent to determine a target model combination corresponding to the environmental data and the work instruction, wherein the multifunctional intelligent agent includes a plurality of machine learning models with different functions, and the target model combination is used to characterize the combination mode and input and output information between the machine learning models. The multifunctional intelligent agent is trained by inverse reinforcement learning based on the historical environmental data after generalization operation and the expert demonstration information corresponding to the historical environmental data, and the expert demonstration information includes: a target model combination corresponding to the historical environmental data set manually;

[0088] The instruction execution module 74 is used to use the target model combination to determine the operation instructions corresponding to the work instructions based on the environmental data, wherein the operation instructions are used to control the multifunctional intelligent agent to perform operations that can meet the task requirements corresponding to the work instructions.

[0089] Optionally, the training steps of the multifunctional intelligent agent include: obtaining historical environmental data in the environment corresponding to the environmental data, and expert demonstration information corresponding to the historical environmental data; using transfer learning to generalize the historical environmental data and expert demonstration information, wherein the generalization operation is used to extract common features from different historical environmental data or expert demonstration information; using inverse reinforcement learning and reinforcement learning to train the multifunctional intelligent agent based on the historical environmental data and expert demonstration information after generalization.

[0090] Optionally, the multifunctional intelligent agent is trained based on the historical environmental data and expert demonstration information after generalization operations by inverse reinforcement learning and reinforcement learning, including: performing inverse reinforcement learning on the historical environmental data and expert demonstration information after generalization operations to obtain a reward function, wherein the reward function is used to characterize the reward value corresponding to the multifunctional intelligent agent performing different predicted actions under different environmental data, and the predicted action is used to determine the target model combination; using the multifunctional intelligent agent to predict the target model combination corresponding to the historical environmental data based on the reward function; determining the error loss between the predicted target model combination and the target model combination corresponding to the expert demonstration information; updating the reward function based on the error loss, and using the multifunctional intelligent agent to repeat the above prediction process based on the updated reward function until the error loss is less than a preset error threshold.

[0091] Optionally, obtaining environmental data and work instructions includes: obtaining multimodal data collected by sensor equipment of the multifunctional intelligent body, wherein the sensor equipment includes at least one of the following: a visual sensor, an audio sensor, and a balancer sensor; determining a data type of each data in the multimodal data, converting the data into a preset standard format corresponding to the data type, and integrating the multimodal data converted into the preset standard format into a data file in a preset format; performing a generalization operation on the data in the data file to obtain environmental data, wherein the generalization operation is used to extract common features from different data.

[0092] Optionally, when the historical environment data or the data file contains image data, the generalization operation includes: using a diffusion model to extract feature values ​​of the image data and determine feature texts corresponding to the feature values, wherein the diffusion model includes at least one of the following: a Stable Diffusion model; determining the part of speech of each word in the feature text, wherein the part of speech includes: noun; determining the common noun combination between the feature texts corresponding to different image data, and obtaining the features of the image data after the generalization operation.

[0093] Optionally, the data processing device is also used to: determine the operation mode of the multifunctional intelligent body, wherein the operation mode includes: a working mode and a learning mode, wherein, when the operation mode is the working mode, the multifunctional intelligent body is used to respond to the work instruction and perform operations that can meet the task requirements corresponding to the work instruction, and when the operation mode is the learning mode, the multifunctional intelligent body is used to perform autonomous learning based on the multimodal data collected by the sensor device.

[0094] Optionally, the multifunctional intelligent body also includes: an operation module, the operation module is used to receive operation instructions and execute operation actions corresponding to the operation instructions, wherein the operation action includes at least one of the following: a robot arm movement action, an audio playback action.

[0095] It should be noted that the various modules in the above-mentioned data processing device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0096] It should be noted that the data processing device provided in this embodiment can be used to execute Figure 2 The data processing method shown, therefore, the relevant explanations and instructions on the above data processing method are also applicable to the embodiments of the present application and will not be repeated here.

[0097] An embodiment of the present application also provides a non-volatile storage medium, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following data processing method or agent training method by running the computer program: obtaining environmental data and work instructions, wherein the environmental data is used to characterize information about the environment in which the multifunctional agent is located; using the multifunctional agent to determine a target model combination corresponding to the environmental data and the work instructions, wherein the multifunctional agent includes multiple machine learning models with different functions, and the target model combination is used to characterize the combination mode and input and output information between the machine learning models, and the multifunctional agent is trained by inverse reinforcement learning based on historical environmental data after generalization operations and expert demonstration information corresponding to the historical environmental data, and the expert demonstration information includes: an artificially set target model combination corresponding to the historical environmental data; using the target model combination, based on the environmental data, determining an operation instruction corresponding to the work instruction, wherein the operation instruction is used to control the multifunctional agent to perform operations that can meet the task requirements corresponding to the work instruction.

[0098] Alternatively, environmental data and expert demonstration information corresponding to the environmental data are obtained, wherein the environmental data is used to characterize information about the environment in which the multifunctional intelligent agent is located, the multifunctional intelligent agent contains multiple machine learning models with different functions, and the expert demonstration information includes: a combination of target models corresponding to the environmental data set artificially, the target model combination is used to characterize the combination mode and input and output information between machine learning models; the environmental data and expert demonstration information are generalized by transfer learning, wherein the generalization operation is used to extract common features from different environmental data or expert demonstration information; the multifunctional intelligent agent is trained by inverse reinforcement learning and reinforcement learning based on the environmental data and expert demonstration information after generalization operation to obtain a target multifunctional intelligent agent, wherein the target multifunctional intelligent agent is used to determine the target model combination corresponding to the environmental data.

[0099] The present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of the data processing method or the steps of the intelligent agent training method described in each embodiment of the present application: obtaining environmental data and work instructions, wherein the environmental data is used to characterize the information of the environment in which the multifunctional intelligent agent is located; using the multifunctional intelligent agent to determine the target model combination corresponding to the environmental data and the work instruction, wherein the multifunctional intelligent agent includes multiple machine learning models with different functions, and the target model combination is used to characterize the combination mode and input and output information between the machine learning models. The multifunctional intelligent agent is trained by inverse reinforcement learning based on historical environmental data after generalization operation and expert demonstration information corresponding to the historical environmental data, and the expert demonstration information includes: an artificially set target model combination corresponding to the historical environmental data; using the target model combination, based on the environmental data, determining the operation instruction corresponding to the work instruction, wherein the operation instruction is used to control the multifunctional intelligent agent to perform operations that can meet the task requirements corresponding to the work instruction.

[0100] Alternatively, environmental data and expert demonstration information corresponding to the environmental data are obtained, wherein the environmental data is used to characterize information about the environment in which the multifunctional intelligent agent is located, the multifunctional intelligent agent contains multiple machine learning models with different functions, and the expert demonstration information includes: a combination of target models corresponding to the environmental data set artificially, the target model combination is used to characterize the combination mode and input and output information between machine learning models; the environmental data and expert demonstration information are generalized by transfer learning, wherein the generalization operation is used to extract common features from different environmental data or expert demonstration information; the multifunctional intelligent agent is trained by inverse reinforcement learning and reinforcement learning based on the environmental data and expert demonstration information after generalization operation to obtain a target multifunctional intelligent agent, wherein the target multifunctional intelligent agent is used to determine the target model combination corresponding to the environmental data.

[0101] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0102] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0104] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0105] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0107] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Acquiring environmental data and work instructions, wherein the environmental data is used to characterize information about the environment in which the multifunctional intelligent body is located; The multifunctional agent is used to determine a target model combination corresponding to the environmental data and the work instruction, wherein the multifunctional agent includes a plurality of machine learning models with different functions, and the target model combination is used to characterize the combination mode and input and output information between the machine learning models. The multifunctional agent is trained by inverse reinforcement learning based on the historical environmental data after generalization operation and the expert demonstration information corresponding to the historical environmental data, and the expert demonstration information includes: the target model combination corresponding to the historical environmental data set artificially; The target model combination is used to determine the operation instruction corresponding to the work instruction based on the environmental data, wherein the operation instruction is used to control the multifunctional intelligent agent to perform an operation that can meet the task requirements corresponding to the work instruction.

2. The data processing method according to claim 1, characterized in that: The training steps of the multifunctional agent include: Acquire the historical environmental data in the environment corresponding to the environmental data, and the expert demonstration information corresponding to the historical environmental data; Performing a generalization operation on the historical environment data and the expert demonstration information by means of transfer learning, wherein the generalization operation is used to extract common features from different historical environment data or expert demonstration information; The multifunctional intelligent agent is trained by adopting the inverse reinforcement learning and reinforcement learning methods according to the historical environment data and the expert demonstration information after the generalization operation.

3. The data processing method according to claim 2, characterized in that: The method of inverse reinforcement learning and reinforcement learning is adopted to train the multifunctional intelligent agent according to the historical environment data and the expert demonstration information after the generalization operation, including: Performing inverse reinforcement learning on the historical environment data and the expert demonstration information after the generalization operation to obtain a reward function, wherein the reward function is used to characterize the reward value corresponding to the multifunctional intelligent agent performing different predicted actions under different environment data, and the predicted actions are used to determine the target model combination; Using the multifunctional agent, based on the reward function, predicting the target model combination corresponding to the historical environment data; Determining an error loss between the predicted target model combination and the target model combination corresponding to the expert demonstration information; According to the error loss, the reward function is updated, and the multifunctional agent is used to repeat the above prediction process according to the updated reward function until the error loss is less than a preset error threshold.

4. The data processing method according to claim 1, characterized in that: Obtaining environmental data and work instructions includes: Acquire multimodal data collected by sensor equipment of the multifunctional intelligent body, wherein the sensor equipment includes at least one of the following: a visual sensor, an audio sensor, and a balance sensor; Determining a data type of each data in the multimodal data, converting the data into a preset standard format corresponding to the data type, and integrating the multimodal data converted into the preset standard format into a data file in a preset format; A generalization operation is performed on the data in the data file to obtain the environmental data, wherein the generalization operation is used to extract common features from different data.

5. The data processing method according to claim 2 or 4, characterized in that: When the historical environment data or the data in the data file includes image data, the generalization operation includes: A diffusion model is used to extract feature values ​​of the image data and determine feature text corresponding to the feature values, wherein the diffusion model includes at least one of the following: a Stable Diffusion model; Determine the part of speech of each participle in the feature text, wherein the part of speech includes: noun; The common noun combination between the feature texts corresponding to the different image data is determined to obtain the features of the image data after the generalization operation.

6. The data processing method according to claim 1, characterized in that: The method further comprises: Determine an operation mode of the multifunctional intelligent agent, wherein the operation mode includes: a working mode and a learning mode, wherein, when the operation mode is the working mode, the multifunctional intelligent agent is used to respond to the work instruction and perform operations that can meet the task requirements corresponding to the work instruction, and when the operation mode is the learning mode, the multifunctional intelligent agent is used to perform autonomous learning based on the multimodal data collected by the sensor device.

7. The data processing method according to claim 6, characterized in that: The multifunctional intelligent body also includes: an operation module, which is used to receive the operation instruction and execute the operation action corresponding to the operation instruction, wherein the operation action includes at least one of the following: a robot arm movement action and an audio playback action.

8. A method for training an intelligent agent, characterized in that: include: Acquire environmental data and expert demonstration information corresponding to the environmental data, wherein the environmental data is used to characterize information about the environment in which the multifunctional intelligent agent is located, the multifunctional intelligent agent includes multiple machine learning models with different functions, and the expert demonstration information includes: a combination of target models corresponding to the environmental data set artificially, the combination of target models is used to characterize the combination mode and input and output information between the machine learning models; Performing a generalization operation on the environmental data and the expert demonstration information by means of transfer learning, wherein the generalization operation is used to extract common features from different environmental data or expert demonstration information; The multifunctional agent is trained according to the environmental data after the generalization operation and the expert demonstration information by using inverse reinforcement learning and reinforcement learning to obtain a target multifunctional agent, wherein the target multifunctional agent is used to determine the target model combination corresponding to the environmental data.

9. A data processing device, characterized in that: include: A data acquisition module, used to acquire environmental data and work instructions, wherein the environmental data is used to characterize information about the environment in which the multifunctional intelligent body is located; A data analysis module, used to use the multifunctional agent to determine a target model combination corresponding to the environmental data and the work instruction, wherein the multifunctional agent includes a plurality of machine learning models with different functions, and the target model combination is used to characterize the combination mode and input and output information between the machine learning models, and the multifunctional agent is trained by inverse reinforcement learning based on the historical environmental data after generalization operation and the expert demonstration information corresponding to the historical environmental data, and the expert demonstration information includes: the target model combination corresponding to the historical environmental data set manually; An instruction execution module is used to adopt the target model combination and determine the operation instruction corresponding to the work instruction based on the environmental data, wherein the operation instruction is used to control the multifunctional intelligent agent to perform an operation that can meet the task requirements corresponding to the work instruction.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program, when running, executes the data processing method described in any one of claims 1 to 7 or the intelligent agent training method described in claim 8.

11. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the data processing method described in any one of claims 1 to 7 or the intelligent agent training method described in claim 8 by running the computer program.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data processing method described in any one of claims 1 to 7 or the steps of the intelligent agent training method described in claim 8 are implemented.

Images