A model training method and related device

Abstract

The present application discloses a model training method and related devices, which are applied to the field of robotics. The model training method includes: obtaining training data, superimposing noise on the training data; using a data generation model to extract principal component information from noisy training data, and reconstructing data based on the principal component information; inputting the reconstructed data into a reinforcement learning model for training; determining the reconstruction loss based on the noisy training data and the reconstructed data, and determining the reinforcement learning loss of the reinforcement learning model; updating the parameters of the data generation model and the reinforcement learning model based on the determined loss; after the training is completed, a data generation model and a reinforcement learning model for deployment on a physical robot are obtained. Through the method disclosed in the present application, a model with strong noise resistance (a reinforcement learning model based on a data generation model) can be quickly trained, and when it is deployed on a physical robot, better results can be obtained in the real world.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a model training method and related devices. Background Art

[0002] With the rapid development of artificial intelligence technology, robotics has gone from science fiction to reality and is widely used in many fields such as industry, family, medical care, education, etc. Robots can not only complete highly repetitive and dangerous labor, but also navigate and operate autonomously in complex environments, greatly improving productivity and convenience of life.

[0003] As a branch of machine learning, reinforcement learning (RL) enables agents to continuously learn and optimize decision-making strategies in the environment through trial-and-error feedback mechanisms (the reinforcement learning model includes agents and the environment, and the agents interact with the environment during the entire reinforcement learning process), thereby achieving autonomous behavior. The application of reinforcement learning in robotics technology enables robots to autonomously complete tasks such as navigation, operation, obstacle avoidance, and collaboration by interacting with the environment, and has a stronger level of adaptability and intelligence.

[0004] Since the cost of training reinforcement learning models on physical robots is high and the hardware loss is huge, most reinforcement learning models are currently trained in a robot simulation environment and then deployed to a physical robot. However, when the reinforcement learning model trained in a robot simulation environment is deployed to a physical robot, the effect of the reinforcement learning model in the real environment is much worse than that in the robot simulation environment. Summary of the invention

[0005] In view of this, the present application provides a model training method and related devices to solve the problem that when a reinforcement learning model trained in a robot simulation environment is deployed to a physical robot, the effect of the reinforcement learning model in a real environment is much worse than that in a robot simulation environment. The technical solution is as follows:

[0006] The first aspect of the present application provides a model training method, comprising:

[0007] Acquire training data, and superimpose noise on the training data to obtain noisy training data;

[0008] Extracting principal component information from the noisy training data using a data generation model, and reconstructing data based on the principal component information to obtain reconstructed data with the same dimension as the noisy training data;

[0009] Inputting the reconstructed data into a reinforcement learning model for training;

[0010] Determine a reconstruction loss of the data generation model according to the noisy training data and the reconstructed data, and determine a reinforcement learning loss of the reinforcement learning model on the reconstructed data;

[0011] The parameters of the data generation model and the reinforcement learning model are updated according to the reconstruction loss and the reinforcement learning loss; after the training is completed, the data generation model and the reinforcement learning model for deployment on the physical robot are obtained.

[0012] In a possible implementation, extracting principal component information from the noisy training data using a data generation model includes:

[0013] Determine the principal component distribution of the noisy training data using a data generation model;

[0014] The data generation model is used to sample random values ​​from a standard normal distribution, and the random values ​​are mapped to the principal component distribution to obtain principal component information.

[0015] In a possible implementation, the data generation model includes an encoder and a decoder;

[0016] The using of the data generation model to determine the principal component distribution of the noisy training data includes:

[0017] Using the encoder of the data generation model, determine the mean and variance that characterize the distribution of the principal components;

[0018] Using the data generation model, data is reconstructed according to the principal component information to obtain reconstructed data with the same dimension as the noisy training data, including:

[0019] The decoder of the data generation model is used to reconstruct data according to the principal component information to obtain reconstructed data with the same dimension as the noisy training data.

[0020] In a possible implementation, the encoder includes a first fully connected layer and a second fully connected layer, and the noisy training data is processed by the first fully connected layer and the second fully connected layer in sequence;

[0021] If N / 2 is an even number, the output dimension M of the first fully connected layer is N / 2; if N / 2 is not an even number, the output dimension M of the first fully connected layer is an even number closest to N / 2, the output dimension of the second fully connected layer is M / 2, and N is the dimension of the noisy training data;

[0022] The decoder includes a third fully connected layer and a fourth fully connected layer, and the principal component information is processed by the third fully connected layer and the fourth fully connected layer in sequence. The output dimension of the third fully connected layer is M, and the output dimension of the fourth fully connected layer is N.

[0023] In a possible implementation, the random value is mapped to the principal component distribution using the data generation model to obtain principal component information, including:

[0024] Multiplying the random value by the variance representing the principal component distribution to obtain a multiplication result;

[0025] The multiplication result is summed with the mean representing the distribution of the principal component to obtain the principal component information.

[0026] In a possible implementation, updating parameters of the data generation model and the reinforcement learning model according to the reconstruction loss and the reinforcement learning loss includes:

[0027] Fusion the reconstruction loss and the reinforcement learning loss to obtain a fused loss;

[0028] According to the post-fusion loss, parameters of the data generation model and the reinforcement learning model are updated.

[0029] A second aspect of the present application provides a model training device, comprising: a training data acquisition module, a noise superposition module, a data generation module, a data input module, a loss determination module and a parameter update module;

[0030] The training data acquisition module is used to acquire training data;

[0031] The noise superposition module is used to superimpose noise on the training data to obtain noisy training data;

[0032] The data generation module is used to extract principal component information from the noisy training data using a data generation model, and reconstruct data according to the principal component information to obtain reconstructed data with the same dimension as the noisy training data;

[0033] The data input module is used to input the reconstructed data into a reinforcement learning model for training;

[0034] The loss determination module is used to determine the reconstruction loss of the data generation model based on the noisy training data and the reconstructed data, and determine the reinforcement learning loss of the reinforcement learning model on the reconstructed data;

[0035] The parameter updating module is used to update the parameters of the data generation model and the reinforcement learning model according to the reconstruction loss and the reinforcement learning loss; after the training is completed, the data generation model and the reinforcement learning model for deployment on the physical robot are obtained.

[0036] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0037] The memory is used to store computer programs;

[0038] The processor is used to execute the computer program so that the electronic device can implement the steps of any one of the model training methods described above.

[0039] The fourth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the above-mentioned model training methods.

[0040] The fifth aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of any one of the above-mentioned model training methods.

[0041] By means of the above technical scheme, the model training method provided by the present application, after obtaining the training data, first superimposes noise on the training data to obtain noisy training data, and then uses the data generation model to extract principal component information from the noisy training data, and reconstructs the data based on the principal component information to obtain reconstructed data with the same dimension as the noisy training data, and then inputs the reconstructed data into the reinforcement learning model for training, and then determines the reconstruction loss of the data generation model based on the noisy training data and the reconstructed data, and determines the reinforcement learning loss of the reinforcement learning model on the reconstructed data, and finally updates the parameters of the data generation model and the reinforcement learning model based on the reconstruction loss and the reinforcement learning loss. Taking into account that noise factor is one of the main factors leading to poor effect of reinforcement learning model in real environment, in order to enable the reinforcement learning model to obtain better effect in real environment when deployed to the physical robot, the present application superimposes noise on the training data, and introduces a data generation model based on the reinforcement learning model. After obtaining the noisy training data, the noisy training data is not directly input into the reinforcement learning model for training. Instead, the noisy training data is first input into the data generation model, and the data generation model is used to extract the principal component information of the noisy training data, and then the data is reconstructed according to the principal component information of the noisy training data. Then, the reconstructed data is input into the reinforcement learning model for training. On the one hand, the data generation model is used to extract the noisy training data, and the data generation model is used to extract the principal component information of the noisy training data. The principal component information indirectly realizes denoising. In addition, since the data generation model does not directly denoise, but achieves indirect denoising through the extraction of principal component information, any type of noise can be removed, and noise with a large fluctuation range can also be effectively removed. On the other hand, inputting the data denoised by the data generation model into the reinforcement learning model for training can greatly reduce the misleading of noise on the exploration stage of reinforcement learning, thereby reducing the training difficulty of the reinforcement learning model and improving the convergence speed of the reinforcement learning model. In summary, through the model training method provided in this application, a model with strong noise resistance (reinforcement learning model based on the data generation model) can be quickly trained, and then when it is deployed to a physical robot, better results can be obtained in the real world. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A schematic diagram of a system architecture involved in this application;

[0044] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;

[0046] Figure 4 A flowchart of a model training method provided in an embodiment of the present application;

[0047] Figure 5 A flow chart of using a data generation model provided in an embodiment of the present application to extract principal component information from noisy training data, and reconstructing data based on the principal component information to obtain reconstructed data with the same dimension as the noisy training data;

[0048] Figure 6 A schematic diagram of the structure of the data generation model provided in the embodiment of the present application;

[0049] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0051] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0053] In one possible implementation, Figure 1As shown, the system architecture involved in the present application may include a terminal 101 and a server 102, and the terminal 101 may interact with the server 102 via a network (wired network or wireless network). The server 102 may include one or more servers ( Figure 1 The terminal 101 may obtain training data, and transmit the obtained training data to the server 102 through the network. The server 102 uses the training data to train the reinforcement learning model according to the model training method provided in this application.

[0054] In another possible implementation, the system architecture involved in the present application may include a terminal. The terminal has strong data processing capabilities. The terminal can obtain training data, and use the training data to train the reinforcement learning model according to the model training method provided in the present application.

[0055] Next, the product form of the above-mentioned terminal is described.

[0056] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a robot, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.

[0057] Figure 2 An optional hardware structure diagram of a terminal is shown.

[0058] refer to Figure 2 As shown, the terminal may include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), an earphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0059] The input unit 230 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 270, and can receive and execute the command sent by the processor 270; the touch signal at least includes the contact point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 231, the input unit 230 can also include other input devices. Specifically, other input devices 232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0060] The display unit 240 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0061] The memory 220 can be used to store instructions and data. The memory 220 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; it provides the processor 270 with hardware, software, and data resources including management of computing and processing equipment, and supports control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0062] The processor 270 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 220 and calling data stored in the memory 220, it executes various functions of the terminal and processes data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 270. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 270 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 220, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0063] Among them, the memory 220 can be used to store software codes related to the model training method, the processor 270 can execute the software codes in the memory 220, and can also schedule other units (such as the above-mentioned input unit 230 and display unit 240) to implement corresponding functions.

[0064] The radio frequency unit 210 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 270 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 210 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0065] In the embodiment of the present application, the radio frequency unit 210 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional, and can be replaced by other communication interfaces, such as a network port.

[0066] The terminal also includes a power supply 290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0067] The terminal also includes an external interface 280, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.

[0068] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0069] Next, the product form of the above server is described.

[0070] Figure 3 A structural diagram of the above server is provided, such as Figure 3 As shown, the server may include a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate with each other via the bus 301.

[0071] The bus 301 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0072] The processor 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0073] The memory 304 may include a volatile memory, such as a random access memory (RAM). The memory 304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0074] The memory 304 can be used to store software codes related to the model training method. The processor 302 can call the software codes stored in the memory 304, and can also schedule other units to implement corresponding functions.

[0075] The processors in the above-mentioned terminals and servers (for example, processor 270 and processor 302) may be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or combinations of these hardware circuits. For example, the processor may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0076] At present, the method of training a reinforcement learning model in a robot simulation environment is to directly input the training data into the reinforcement learning model for training. However, when the reinforcement learning model trained by the above method is deployed to a physical robot, the effect of the reinforcement learning model in the real environment is much worse than that in the robot simulation environment. In response to the above problem, the inventor of this case conducted a study and found that the noise factor is one of the main factors leading to the poor effect of the reinforcement learning model in the real environment. Based on this finding, the inventor of this case thought of adding noise to the training data to obtain noisy training data, and then inputting the noisy training data into the reinforcement learning model for training, in the hope that the reinforcement learning model can learn the ability to resist noise, thereby reducing the loss of effect in the real world.

[0077] The inventors of this case studied the above-mentioned method of inputting noisy training data into the reinforcement learning model for training and found that the method of inputting noisy training data into the reinforcement learning model for training has the following problems: First, the characteristics of reinforcement learning itself cause noise to greatly mislead the exploration stage during training. Although a large amount of data can be used to statistically enable the reinforcement learning model to learn the required capabilities, it will undoubtedly greatly increase the training difficulty and convergence time of the reinforcement learning model; second, because the noise in the training data will directly affect the results of the reinforcement learning model exploration, the noise is generally controlled within a relatively small range, which makes the reinforcement learning model's ability to resist noise limited.

[0078] In view of the problems existing in the above-mentioned method of inputting noisy training data into the reinforcement learning model for training, the inventor of this case continued to conduct research and finally proposed a model training method with better effect through continuous research. Next, the model training method provided by this application is introduced through the following embodiments.

[0079] See also Figure 4 , shows a flow chart of a model training method provided in an embodiment of the present application, and the model training method may include:

[0080] Step S401: Acquire training data, and superimpose noise on the training data to obtain noisy training data.

[0081] Training data is obtained from a training data set, and random noise is superimposed on the obtained training data to obtain noisy training data.

[0082] Considering that noise is one of the main factors that lead to poor performance of reinforcement learning models in real environments, after obtaining training data, this embodiment superimposes noise on the training data to obtain noisy training data, and then uses the noisy training data to perform model training.

[0083] Step S402: extracting principal component information from the noisy training data using the data generation model, and reconstructing the data based on the principal component information to obtain reconstructed data with the same dimension as the noisy training data.

[0084] This embodiment performs model training in a robot simulation environment.

[0085] In order to effectively improve the application effect of the reinforcement learning model in a real environment and reduce the learning difficulty and convergence time of the reinforcement learning model, the embodiment of the present application introduces a data generation model.

[0086] After obtaining the noisy training data, the data generation model is used to extract the principal component information from the noisy training data, and the data is reconstructed based on the principal component information to obtain the reconstructed data with the same dimension as the noisy training data.

[0087] It should be noted that the principal component information extracted from the noisy training data using the data generation model is the important information in the noisy training data. After obtaining the important information in the noisy training data, the data generation model is further used to reconstruct the data based on the obtained important information to obtain reconstructed data with the same dimension as the noisy training data.

[0088] When using the data generation model to extract important information from noisy training data, non-important information will be filtered out. In addition to the non-important information in the training data, the non-important information also includes noise. In other words, by using the data generation model to extract important information, denoising is indirectly achieved. Therefore, the reconstructed data finally obtained is denoised data.

[0089] Step S403: input the reconstructed data into the reinforcement learning model for training.

[0090] Since the data generation model can remove noise from noisy training data, the introduction of the data generation model greatly reduces the misleading effect of noise on the exploration phase of reinforcement learning, thereby reducing the training difficulty of the reinforcement learning model and improving the convergence speed of the reinforcement learning model.

[0091] Step S404: Determine the reconstruction loss of the data generation model based on the noisy training data and the reconstructed data, and determine the reinforcement learning loss of the reinforcement learning model on the reconstructed data.

[0092] Among them, the reconstruction loss of the data generation model can characterize the difference between the reconstructed data and the noisy training data.

[0093] In one possible implementation, the reconstruction loss L of the data generation model is restruction It can be calculated as follows:

[0094] (1)

[0095] Among them, D i ′ Represents the i-th dimension of the reconstructed data, D i represents the i-th dimension data of the noisy training data, and N represents the dimension of the reconstructed data and the noisy training data.

[0096] It should be noted that this embodiment is not limited to the method of formula (1) to determine the reconstruction loss of the data generation model. Other methods for determining the loss that can determine the difference between the reconstructed data and the noisy training data are also applicable to this application.

[0097] It should also be noted that there are many types of reinforcement learning models, such as value function-based reinforcement learning models, strategy-based reinforcement learning models, etc. The reinforcement learning model in this embodiment can be any type of reinforcement learning model. For any type of reinforcement learning model, the reinforcement learning loss can be determined by using a loss determination method suitable for that type of reinforcement learning model.

[0098] Step S405: Update the parameters of the data generation model and the reinforcement learning model according to the reconstruction loss and the reinforcement learning loss.

[0099] Specifically, the reconstruction loss and the reinforcement learning loss are fused to obtain the fused loss, and the parameters of the data generation model and the reinforcement learning model are updated according to the fused loss.

[0100] In one possible implementation, the reconstruction loss and reinforcement learning loss can be fused by direct summation. If the reconstruction loss is expressed as L restruction , denote the reinforcement learning loss as L RL , then the loss after fusion is L toal It can be expressed as:

[0101] L toal =L restruction +L RL (2)

[0102] It should be noted that this embodiment is not limited to the direct summation method to merge the reconstruction loss and the reinforcement learning loss. For example, the reconstruction loss L can also be set restruction and reinforcement learning loss L RL The corresponding weights are used to merge the reconstruction loss and the reinforcement learning loss by weighted summation.

[0103] The training data in the training data set is used to perform multiple iterations of training according to the process of step S401 to step S405 above, until the training end condition (such as model convergence) is met.

[0104] After the training, a data generation model and a reinforcement learning model are obtained for deployment on a physical robot, that is, a reinforcement learning model based on the data generation model. The trained data generation model can effectively remove noise in the real environment, so that the reinforcement learning model has better effect in the real environment.

[0105] The model training method provided in the embodiment of the present application, after obtaining the training data, first superimposes noise on the training data to obtain noisy training data, then uses the data generation model to extract principal component information from the noisy training data, and reconstructs the data based on the principal component information to obtain reconstructed data with the same dimension as the noisy training data, then inputs the reconstructed data into the reinforcement learning model for training, and then determines the reconstruction loss of the data generation model based on the noisy training data and the reconstructed data, and determines the reinforcement learning loss of the reinforcement learning model on the reconstructed data, and finally updates the parameters of the data generation model and the reinforcement learning model based on the reconstruction loss and the reinforcement learning loss. Taking into account that noise factor is one of the main factors leading to poor effect of reinforcement learning model in real environment, in order to enable the reinforcement learning model to obtain better effect in real environment when deployed to the physical robot, the embodiment of the present application superimposes noise on the training data, and introduces a data generation model based on the reinforcement learning model. After obtaining the noisy training data, the noisy training data is not directly input into the reinforcement learning model for training, but the noisy training data is first input into the data generation model, and the data generation model is used to extract the principal component information of the noisy training data, and then the data is reconstructed according to the principal component information of the noisy training data, and then the reconstructed data is input into the reinforcement learning model for training. On the one hand, the data generation model is used to generate the noise for the noisy training data. Extracting the principal component information indirectly achieves denoising. In addition, since the data generation model does not directly denoise, but achieves indirect denoising through the extraction of principal component information, any type of noise can be removed, and noise with a large fluctuation range can also be effectively removed. On the other hand, inputting the data denoised by the data generation model into the reinforcement learning model can greatly reduce the misleading of noise on the exploration stage of reinforcement learning, thereby reducing the training difficulty of the reinforcement learning model and improving the convergence speed of the reinforcement learning model. In summary, through the model training method provided in the embodiment of the present application, a model with strong noise resistance (reinforcement learning model based on the data generation model) can be quickly trained, and then when it is deployed to a physical robot, better results can be obtained in the real world.

[0106] In another embodiment of the present application, the specific implementation process of "Step S102: using the data generation model to extract principal component information from the noisy training data, and reconstructing the data based on the principal component information to obtain reconstructed data with the same dimension as the noisy training data" in the above embodiment is introduced.

[0107] like Figure 5 As shown, the process of extracting principal component information from noisy training data using a data generation model and reconstructing data based on the principal component information to obtain reconstructed data with the same dimension as the noisy training data may include:

[0108] Step S501: using the data generation model to determine the principal component distribution of noisy training data.

[0109] The principal component distribution of the noisy training data is the probability distribution obeyed by the principal component information of the noisy training data.

[0110] In one possible implementation, the data generation model may include an encoder, such as Figure 6 As shown, the noisy training data is input into the encoder of the data generation model. The encoder encodes the noisy training data into a probability distribution obeyed by the principal component information in the latent space, namely, the principal component distribution, and outputs the mean μ and variance σ representing the principal component distribution.

[0111] In a possible implementation, the encoder of the data generation model may include a first fully connected layer and a second fully connected layer, and the noisy training data is processed by the first fully connected layer and the second fully connected layer in sequence. Assuming that the dimension of the noisy training data is N, if N / 2 is an even number, the output dimension M of the first fully connected layer is N / 2, and if N / 2 is not an even number, the output dimension M of the first fully connected layer is an even number closest to N / 2, and the output dimension of the second fully connected layer is M / 2. Exemplarily, the dimension of the noisy training data is 20, the dimension of the data output after processing by the first fully connected layer is 10, and the dimension of the data output after processing by the second fully connected layer is 5.

[0112] Step S502: using the data generation model, sampling from the standard normal distribution to obtain random values, and mapping the random values ​​to the principal component distribution to obtain principal component information.

[0113] After determining the principal component distribution, this embodiment further uses the data generation model to sample a random value ε from the standard normal distribution N (0, 1), and then maps the random value ε to the principal component distribution to obtain principal component information.

[0114] Specifically, using the data generation model, the random value ε is mapped to the principal component distribution, and the process of obtaining the principal component information may include: multiplying the random value ε with the variance σ representing the principal component distribution to obtain the multiplication result εσ; summing the multiplication result εσ with the mean μ representing the principal component distribution to obtain the principal component information Z. That is, the principal component information Z is:

[0115] Z = μ + εσ (3)

[0116] Step S503: using the data generation model, reconstructing the data according to the principal component information to obtain reconstructed data with the same dimension as the noisy training data.

[0117] Since the data generation model indirectly achieves denoising when extracting the principal component information, the reconstructed data finally obtained by reconstructing the data based on the principal component information is denoised data.

[0118] like Figure 6 As shown, the data generation model includes not only an encoder but also a decoder. The principal component information obtained through step S502 can be further input into the decoder of the data generation model. The decoder reconstructs the data based on the input principal component information and outputs reconstructed data with the same dimension as the noisy training data.

[0119] Assuming that the encoder of the data generation model includes a first fully connected layer and a second fully connected layer, the encoder of the data generation model includes a third fully connected layer and a fourth fully connected layer. The principal component information obtained by step S502 is processed by the third fully connected layer and the fourth fully connected layer in sequence, and finally reconstructed data with the same dimension as the noisy training data is output, wherein the output dimension of the third fully connected layer is M, and the output dimension of the fourth fully connected layer is N.

[0120] It should be noted that this embodiment does not limit the number of fully connected layers of the encoder and decoder of the data generation model to two. The number of fully connected layers of the encoder and decoder of the data generation model can be determined according to the actual application scenario. For example, in a certain application scenario, the number of fully connected layers of the encoder and decoder of the data generation model is 3.

[0121] After obtaining the reconstructed data (i.e., denoised data) through the above process, the reconstructed data can be input into the reinforcement learning model for training.

[0122] The above introduces the model training method provided in the embodiment of the present application, and the device corresponding to the above model training method will be introduced below.

[0123] See also Figure 7 , Figure 7 A structural schematic diagram of a training device for a reinforcement learning model provided in an embodiment of the present application, the training device for the reinforcement learning model may include: a training data acquisition module 701, a noise superposition module 702, a data generation module 703, a data input module 704, a loss determination module 705 and a parameter update module 706.

[0124] The training data acquisition module 701 is used to acquire training data.

[0125] The noise superposition module 702 is used to superimpose noise on the training data to obtain noisy training data.

[0126] The data generation module 703 is used to extract principal component information from the noisy training data using the data generation model, and reconstruct the data based on the principal component information to obtain reconstructed data with the same dimension as the noisy training data.

[0127] The data input module 704 is used to input the reconstructed data into the reinforcement learning model for training.

[0128] The loss determination module 705 is used to determine the reconstruction loss of the data generation model based on the noisy training data and the reconstructed data, and determine the reinforcement learning loss of the reinforcement learning model on the reconstructed data.

[0129] The parameter updating module 706 is used to update the parameters of the data generation model and the reinforcement learning model according to the reconstruction loss and the reinforcement learning loss; after the training is completed, a reinforcement learning model based on the data generation model is obtained for deployment on the physical robot.

[0130] In a possible implementation, when the data generation module 703 uses the data generation model to extract the principal component information from the noisy training data, it is specifically used to:

[0131] Using the data generation model, determine the principal component distribution of the noisy training data;

[0132] Using the data generation model, random values ​​are sampled from the standard normal distribution, and the random values ​​are mapped to the principal component distribution to obtain the principal component information.

[0133] In one possible implementation, the data generation model includes an encoder and a decoder.

[0134] When the data generation module 703 uses the data generation model to determine the principal component distribution of the noisy training data, it is specifically used to:

[0135] Using the encoder of the data generation model, determine the mean and variance that characterize the distribution of the principal components;

[0136] Using the data generation model, data is reconstructed based on the principal component information to obtain reconstructed data with the same dimension as the noisy training data, including:

[0137] The decoder of the data generation model is used to reconstruct the data according to the principal component information to obtain reconstructed data with the same dimension as the noisy training data.

[0138] In a possible implementation, the encoder of the data generation model includes a first fully connected layer and a second fully connected layer, and the noisy training data is processed by the first fully connected layer and the second fully connected layer in sequence. If N / 2 is an even number, the output dimension M of the first fully connected layer is N / 2. If N / 2 is not an even number, the output dimension M of the first fully connected layer is the even number closest to N / 2, and the output dimension of the second fully connected layer is M / 2, where N is the dimension of the noisy training data.

[0139] The decoder of the data generation model includes a third fully connected layer and a fourth fully connected layer. The principal component information is processed by the third fully connected layer and the fourth fully connected layer in sequence. The output dimension of the third fully connected layer is M, and the output dimension of the fourth fully connected layer is N.

[0140] In a possible implementation, the data generation module 703 uses the data generation model to map the random value to the principal component distribution to obtain the principal component information, specifically for:

[0141] Multiply the random value by the variance representing the distribution of the principal component to obtain the multiplication result;

[0142] The multiplication result is summed with the mean representing the distribution of the principal component to obtain the principal component information.

[0143] In a possible implementation, when the loss determination module 705 updates the parameters of the data generation model and the reinforcement learning model according to the reconstruction loss and the reinforcement learning loss, it is specifically used to:

[0144] Merge the reconstruction loss with the reinforcement learning loss to obtain the fused loss;

[0145] According to the post-fusion loss, parameters of the data generation model and the reinforcement learning model are updated.

[0146] Taking into account that noise is one of the main factors that lead to poor effects of reinforcement learning models in real environments, in order to enable the reinforcement learning model to achieve better results in real environments when it is deployed on a physical robot, the model training device provided in the embodiment of the present application superimposes noise on the training data, and introduces a data generation model based on the reinforcement learning model. After obtaining the noisy training data, the noisy training data is not directly input into the reinforcement learning model for training. Instead, the noisy training data is first input into the data generation model, and the data generation model is used to extract the principal component information of the noisy training data, and then the data is reconstructed according to the principal component information of the noisy training data. Then, the reconstructed data is input into the reinforcement learning model for training. On the one hand, the data generation model is used to train the noisy training data. The principal component information extracted from the training data indirectly realizes denoising. In addition, since the data generation model does not directly denoise, but realizes indirect denoising through the extraction of principal component information, any type of noise can be removed, and noise with a large fluctuation range can also be effectively removed. On the other hand, inputting the data denoised by the data generation model into the reinforcement learning model can greatly reduce the misleading of noise to the exploration stage of reinforcement learning, thereby reducing the training difficulty of the reinforcement learning model and improving the convergence speed of the reinforcement learning model. In summary, the model training device provided in the embodiment of the present application can be used to quickly train a model with strong noise resistance (reinforcement learning model based on the data generation model), and then when it is deployed to a physical robot, better results can be obtained in the real world.

[0147] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.

[0148] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0149] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;

[0150] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0151] Among them, the memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement the steps of the model training method provided in the above embodiment.

[0152] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the model training method provided in the above embodiment.

[0153] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the model training method provided in the above embodiment.

[0154] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0155] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0156] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0157] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A model training method, characterized in that: include: Acquire training data, and superimpose noise on the training data to obtain noisy training data; Extracting principal component information from the noisy training data using a data generation model, and reconstructing data based on the principal component information to obtain reconstructed data with the same dimension as the noisy training data; Inputting the reconstructed data into a reinforcement learning model for training; Determine a reconstruction loss of the data generation model according to the noisy training data and the reconstructed data, and determine a reinforcement learning loss of the reinforcement learning model on the reconstructed data; The parameters of the data generation model and the reinforcement learning model are updated according to the reconstruction loss and the reinforcement learning loss; after the training is completed, the data generation model and the reinforcement learning model for deployment on the physical robot are obtained.

2. The model training method according to claim 1, characterized in that: The data generation model is used to extract principal component information from the noisy training data, including: Determine the principal component distribution of the noisy training data using a data generation model; The data generation model is used to sample random values ​​from a standard normal distribution, and the random values ​​are mapped to the principal component distribution to obtain principal component information.

3. The model training method according to claim 2, characterized in that: The data generation model includes an encoder and a decoder; The using of the data generation model to determine the principal component distribution of the noisy training data includes: Using the encoder of the data generation model, determine the mean and variance that characterize the distribution of the principal components; Using the data generation model, data is reconstructed according to the principal component information to obtain reconstructed data with the same dimension as the noisy training data, including: The decoder of the data generation model is used to reconstruct data according to the principal component information to obtain reconstructed data with the same dimension as the noisy training data.

4. The model training method according to claim 3, characterized in that: The encoder comprises a first fully connected layer and a second fully connected layer, and the noisy training data is processed by the first fully connected layer and the second fully connected layer in sequence; If N / 2 is an even number, the output dimension M of the first fully connected layer is N / 2; if N / 2 is not an even number, the output dimension M of the first fully connected layer is an even number closest to N / 2, the output dimension of the second fully connected layer is M / 2, and N is the dimension of the noisy training data; The decoder includes a third fully connected layer and a fourth fully connected layer, and the principal component information is processed by the third fully connected layer and the fourth fully connected layer in sequence. The output dimension of the third fully connected layer is M, and the output dimension of the fourth fully connected layer is N.

5. The model training method according to claim 3, characterized in that: Using the data generation model, the random value is mapped to the principal component distribution to obtain principal component information, including: Multiplying the random value by the variance representing the principal component distribution to obtain a multiplication result; The multiplication result is summed with the mean representing the distribution of the principal component to obtain the principal component information.

6. The model training method according to claim 1, characterized in that: The updating of parameters of the data generation model and the reinforcement learning model according to the reconstruction loss and the reinforcement learning loss includes: Fusion the reconstruction loss and the reinforcement learning loss to obtain a fused loss; According to the post-fusion loss, parameters of the data generation model and the reinforcement learning model are updated.

7. A model training device, characterized in that: include: Training data acquisition module, noise superposition module, data generation module, data input module, loss determination module and parameter update module; The training data acquisition module is used to acquire training data; The noise superposition module is used to superimpose noise on the training data to obtain noisy training data; The data generation module is used to extract principal component information from the noisy training data using a data generation model, and reconstruct data according to the principal component information to obtain reconstructed data with the same dimension as the noisy training data; The data input module is used to input the reconstructed data into a reinforcement learning model for training; The loss determination module is used to determine the reconstruction loss of the data generation model based on the noisy training data and the reconstructed data, and determine the reinforcement learning loss of the reinforcement learning model on the reconstructed data; The parameter updating module is used to update the parameters of the data generation model and the reinforcement learning model according to the reconstruction loss and the reinforcement learning loss; after the training is completed, the data generation model and the reinforcement learning model for deployment on the physical robot are obtained.

8. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the steps of the model training method as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the model training method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that It includes computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the steps of the model training method as described in any one of claims 1 to 6.

Images