Perception prediction method for cross-modal self-supervised learning for embodied intelligent robots

Abstract

The present invention relates to a perception prediction method for cross-modal self-supervised learning of embodied intelligent robots, which obtains the motion input vector and perception input vector of the robot and uses a cognitive model to learn the correlation between the two. The learning process includes: obtaining the motion input vector and perception input vector related to the robot's behavior, calculating the reconstruction loss corresponding to the motion input vector and the perception input vector by self-encoding and decoding respectively; calculating the cross-modal mapping loss through cross-modal query based on the motion and perception input vectors after self-encoding; based on the reconstruction loss corresponding to the motion perception input vector and the cross-modal mapping loss, the total loss is obtained by weighting, and the parameters of the cognitive model are updated based on the total loss, so as to realize the implicit expression of the modality and the modeling of the connection between the modalities based on self-supervised learning. Compared with the prior art, the present invention can effectively improve the adaptive level of the behavior of the embodied intelligent robot, and has the advantages of autonomous development and continuous learning.

Description

Technical Field

[0001] The present invention relates to the fields of robotics and embodied intelligent technology, and in particular to a perception prediction method for cross-modal self-supervised learning of embodied intelligent robots. Background Art

[0002] With the development of robotics and artificial intelligence technology, machines have gradually evolved from pre-programmed automated systems to perceptual intelligent systems, with the ability to adjust behavior based on environmental perception, and even the ability to predict human intentions. However, they still lack the ability to handle emergencies in open dynamic environments and close collaboration scenarios, and their autonomy and efficiency still need to be improved. Therefore, one of the urgent problems to be solved in embodied intelligence is how robots can sense, understand and reason about their embodied physical interactions with the world. For example, in the field of home services, when service robots face close physical interaction tasks such as double-arm hugging and dexterous grasping, understanding their own behavior is of great significance to ensuring execution efficiency and service quality. However, due to the differences in cross-modal information and the difficulty of self-supervised learning, it is extremely challenging for robots to autonomously understand their own behavior and predict the perceived consequences.

[0003] In order to solve the problem of robot behavior understanding and perception consequence prediction, the cognitive developmental robotics method is an interdisciplinary method inspired by the development mechanism of natural cognitive systems. At present, the research on cognitive developmental robotics in the field of behavior understanding is still in its infancy. In addition, embodied intelligence usually requires robots to have the ability to develop autonomously and learn quickly in order to adapt to the dynamic uncertainty of the environment in real time. Therefore, how to reduce the method's dependence on large amounts of data, enhance its autonomous development capabilities, and at the same time improve the task versatility of robot behavior understanding and perception consequence prediction has become a problem that needs to be solved in this field. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a perception prediction method for cross-modal self-supervised learning of embodied intelligent robots, so as to solve or partially solve the problem of the unsatisfactory sensitivity of embodied intelligent robots to cross-modal features.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] In one aspect of the present invention, a perception prediction method for cross-modal self-supervised learning of an embodied intelligent robot is provided, wherein the motion input vector and the perception input vector of the robot are obtained and used as inputs of a cognitive model to obtain the corresponding perception vector and motion vector after reconstruction, wherein the training process of the cognitive model comprises the following steps:

[0007] Step S1, obtaining a motion input vector and a perception input vector related to the robot behavior, and calculating the reconstruction loss corresponding to the motion input vector and the perception input vector by performing self-encoding and decoding respectively;

[0008] Step S2, calculating a cross-modal mapping loss through cross-modal query based on the motion input vector and the perception input vector after the autoencoder;

[0009] Step S3, based on the reconstruction loss corresponding to the motion input vector and the perception input vector, and the cross-modal mapping loss, a total loss is weighted, and the parameters of the cognitive model are updated based on the total loss to achieve modeling of implicit intra-modal expression and inter-modal connections based on self-supervised learning.

[0010] As a preferred technical solution, in the step S1, the motion input vector and the perceptual input vector are self-encoded using a motion encoder and a perceptual encoder, and the self-encoded motion input vector and the perceptual input vector are decoded using a motion decoder and a perceptual decoder, respectively, wherein the motion encoder includes a plurality of fully connected layers, the motion decoder includes a plurality of fully connected layers, the perceptual encoder includes a plurality of convolutional layers, and the perceptual decoder includes a plurality of deconvolutional layers.

[0011] As a preferred technical solution, in step S1, the reconstruction loss L corresponding to the motion input vector and the perception input vector is reconstruct for:

[0012]

[0013] Where h(·) represents the encoder, g(·) represents the decoder, and X is the motion input vector or the perceptual input vector.

[0014] As a preferred technical solution, in step S2, the cross-modal mapping loss L mapping for:

[0015]

[0016] Among them, F m 、F p are the motion input vector and the perception input vector after self-encoding, Represents a cross-modal query network.

[0017] As a preferred technical solution, in step S3, the total loss L is:

[0018] L=L reconstruct_motion +L reconstruct_perception +β×L mapping

[0019] Among them, Lreconstruct_motion , L reconstruct_perception are the reconstruction losses corresponding to the motion input vector and the perception input vector, respectively, mapping is the cross-modal mapping loss, and β is the weighting coefficient.

[0020] As a preferred technical solution, in the step S3, based on the total loss, gradient descent back propagation is used to update the parameters of the cognitive model based on Adam optimization to achieve training of the cognitive model.

[0021] As a preferred technical solution, the motion input vector includes angle and velocity information of robot joints related to robot behavior.

[0022] As a preferred technical solution, the perception input vector includes contact pressure information on the surface of the robot's mechanical arm and information on the proximity distance to the obstacle related to the robot's behavior.

[0023] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned perception prediction method for cross-modal self-supervised learning for embodied intelligent robots.

[0024] Another aspect of the present invention provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned perception prediction method for cross-modal self-supervised learning for embodied intelligent robots.

[0025] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0026] (1) Association between robot embodied behavior and perceived consequences: The present invention uses implicit intra-modal expression and cross-modal learning to train a neural network for behavior-related motion and perception data, and obtains a query-driven cognitive model that enables the robot to quickly and effectively predict the perceived consequences of its embodied behavior.

[0027] (2) Autonomous and continuous learning: The present invention realizes autonomous modeling of robot embodied behavior based on neural networks. The cross-modal self-supervised learning method can reduce the labeling requirements and quantity dependence of the neural network model on the training data samples, and the performance of the method can be autonomously developed through subsequent embodied behaviors, thereby enhancing the robot's ability to continuously learn.

[0028] (3) Adaptability to unknown environmental conditions: The present invention is based on a cross-modal learning method. It learns how to interact with the environment through sensory-motor experience, enabling the robot to learn body perception behavior patterns, enhance its adaptability when facing unknown environmental conditions, improve its versatility in unfamiliar tasks, and has the potential to achieve rapid re-learning of new tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of a perception prediction method for cross-modal self-supervised learning of an embodied intelligent robot in an embodiment;

[0030] Figure 2 This is a neural network model architecture diagram in the embodiment;

[0031] Figure 3 A neural network structure diagram provided in an embodiment;

[0032] Figure 4 Schematic diagram of the self-supervised learning data flow provided in the embodiment. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] Example 1

[0035] In response to the problems existing in the above-mentioned prior art, this embodiment provides a perception prediction method for cross-modal self-supervised learning of embodied intelligent robots, in which the robot's embodied behavior mainly refers to physical interaction behaviors involving close contact of limbs. This method models the robot's behavior based on a neural network model, effectively predicts the corresponding perception consequences based on motion data, and realizes the learning of the robot's body perception behavior pattern; the use of cross-modal self-supervised learning methods reduces the requirements for data samples, improves the method's autonomous development potential, and enhances the robot's ability to continuously learn to adapt to the dynamic uncertainty in close physical interactions.

[0036] like Figure 1 As shown, the method comprises the following steps:

[0037] Step S1, based on the robot's embodied behavior, obtain relevant motion data.

[0038] For robot embodied behaviors in physical interaction scenarios involving close contact between limbs, the focus is on the robotic arm, and the robotic arm joint information is used as the motion space. In order to obtain effective and accurate data, the angles and velocities of all six joints are obtained based on the joint encoders equipped with the robotic arm.

[0039] Step S2, based on the robot's embodied behavior, obtain relevant perception data.

[0040] For robot embodied behavior in physical interaction scenarios involving close contact between limbs, the focus is on the robotic arm, using close-scale tactile sensing as the perception space. In order to obtain effective and accurate data, the robotic arm is completely covered with distributed multi-sensor electronic skin, which can obtain the contact pressure and obstacle approach distance of 7×15=105 points on the surface of the complete robotic arm.

[0041] In this embodiment, the robot arm is controlled to interact closely with objects or people in the surrounding environment, and all sensor data are transmitted to the host computer based on wireless communication. The motion data and perception data related to the embodied behavior are collected one-to-one. The data in the modality involves multiple sensors. Considering the real-time requirements of close physical interaction, the sampling frequency is set to 10Hz. In this embodiment, each close physical interaction strategy repeats the sampling 30 times to ensure that the sensor output value covers the range as much as possible, and at the same time, the comprehensiveness of sensor data acquisition is ensured as much as possible by adjusting the interactive action. In addition, in order to ensure the authenticity and validity of the data, before data collection, a verified high-precision table conversion measurement tool is used to provide a reliable benchmark value and calibrate the signal model of all sensors.

[0042] Step S3, based on the motion and perception data related to the robot's behavior, a weighted loss function is used to train the cognitive model to learn the query-driven implicit expression of motion and perception, so as to predict the perceptual consequences of embodied behavior and complete the robot's understanding of its own behavior; wherein, the acquisition process of the cognitive model includes implicit expression within the modality based on self-supervised learning and modeling of the connection between modalities.

[0043] This step specifically includes the following sub-steps:

[0044] Step S301, based on a deep learning framework, a neural network model is constructed based on the motion data related to the robot's behavior and its corresponding perception data.

[0045] Based on the joint information related to the robot's behavior and its corresponding tactile information, a neural network model is constructed, such as Figure 2 As shown, the model includes an intra-modal autoencoder and a cross-modal query network module.

[0046] Step S302, based on the intra-modal autoencoder in the neural network model, obtain implicit coding features corresponding to motion and perception respectively. Specifically, this step includes the following sub-steps:

[0047] S3021, obtaining motion (perception) data as input data, and building an intra-modal autoencoder, including a motion (perception) encoder and a motion (perception) decoder;

[0048] like Figure 3 As shown, the motion encoder h m It consists of 3 fully connected layers, where the first layer has an input dimension of 6 and an output dimension of 32, the second layer has an input dimension of 32 and an output dimension of 32, and the third layer has an input dimension of 32 and an output dimension of 10. The corresponding motion decoder g m It consists of 3 fully connected layers, where the input dimension of the first layer is 10 and the output dimension is 32, the input dimension of the second layer is 32 and the output dimension is 32, and the input dimension of the third layer is 32 and the output dimension is 6.

[0049] like Figure 3 As shown, the perceptual encoder h p It consists of 3 convolutional layers, where the first layer has an input dimension of 2, an output dimension of 16, and a 5×5 convolution kernel; the second layer has an input dimension of 16, an output dimension of 16, and a 3×3 convolution kernel; the third layer has an input dimension of 16, an output dimension of 10, and a 1×9 convolution kernel. The corresponding motion decoder g p It consists of 3 deconvolution layers, where the first layer has an input dimension of 10, an output dimension of 16, and a 1×9 deconvolution kernel; the second layer has an input dimension of 16, an output dimension of 16, and a 3×3 deconvolution kernel; the third layer has an input dimension of 16, an output dimension of 2, and a 5×5 deconvolution kernel.

[0050] S3022, define the loss function of the intra-modal autoencoder; for both the motion autoencoder and the perceptual autoencoder, the root mean square error is used as the loss function.

[0051] S3023, based on the motion (perception) autoencoder, obtain the modal implicit coding features of the input data; specifically, given the joint (tactile) input space X m (X p ) and joint (tactile) feature space F m (F p ), for the motion (perceptual) encoder h m (h p ) and motion (perceptual) decoder g m (g p )have:

[0052] h m : X m →Fm ; g m : F m →X m (1)

[0053] h p : X p →F p ; g p : F p →X p (2)

[0054] S3024, forwardly calculate the loss function based on the input data and the data decoded from the implicitly encoded features in the modality. Specifically, for a single modality (motion or perception), given input X∈X, there is a reconstruction loss function L reconstruct It is expressed as:

[0055] L reconstruct =argmin h,g ‖Xg[h(X)]‖ 2 (3)

[0056] In the formula, h(·) represents the encoder, g(·) represents the decoder, and based on the loss function L reconstruct It can measure the completeness of the features extracted by the intra-modal autoencoder.

[0057] Step S303, based on the cross-modal query network module in the neural network model, using the implicit coding features in the modalities of motion and perception, cross-modal implicit expression learning is performed. Specifically, this step includes the following sub-steps:

[0058] S3031, obtains implicit coding features of motion and perception, and builds a cross-modal query network module; it consists of a 3-layer fully connected network, where the first layer has an input dimension of 10 and an output dimension of 32, the second layer has an input dimension of 32 and an output dimension of 32, and the third layer has an input dimension of 32 and an output dimension of 10.

[0059] S3032, define the loss function of cross-modal learning; select the root mean square error as the loss function.

[0060] S3033, forward calculate the loss function based on the implicit coding features of motion and perception. Specifically, given the motion feature input F m ∈F m and the perceptual feature input F p ∈F p , then there is a cross-modal mapping loss function L mappimg It is expressed as:

[0061]

[0062] In the formula, Represents a cross-modal query network based on the loss function L mapping The mapping of motion features to perceptual features can be measured.

[0063] Step S304: train the neural network model using a weighted loss function to obtain a trained cognitive model. Specifically, this step includes the following sub-steps:

[0064] S3041, initialize the neural network model parameters and input the input vector into the model; the initial parameters are generated by a random number seed. The motion input vector consists of the angles and velocities of the six joints, and the perception input vector consists of the contact pressure at the 105 point positions and the approach distance of the obstacle;

[0065] S3042, obtain the corresponding loss function through the intra-modal autoencoder; specifically, input the motion input vector into the motion autoencoder, and forward propagate to calculate the loss function L reconstruct_motion , input the perceptual input vector into the perceptual autoencoder, and forward propagate to calculate the loss function L reconstruct_perception .

[0066] S3043, obtaining a corresponding loss function through a cross-modal query network module; specifically, inputting the motion features and the perceptual features (i.e., the motion input vector and the perceptual input vector after the autoencoder) into the cross-modal query network, and forward propagating to calculate the loss function L mapping .

[0067] S3044, based on the weighted loss function, adjusting and optimizing the current neural network model parameters; specifically, the loss function is the weighted sum of the loss functions of steps S3042 and S3043, which can be expressed as:

[0068] L=L reconstruct_motion +L reconstruct_perception +β×L mapping (5)

[0069] Wherein, β is a weighting coefficient, which can be adjusted according to the category of the robot's embodied behavior (generally related to the task). L represents the total loss function. According to the loss function, the parameters of the entire neural network model are updated based on the Adam optimizer using gradient descent back propagation; in other embodiments, other suitable optimization algorithms can be selected for model training;

[0070] S3045, repeat steps S3041 to S3044 for a preset number of times to obtain the final neural network model parameters. Among them, a test is performed every ten cycles. If the updated model parameters are not further optimized, the current learning rate and batch size are halved, and further training and optimization are performed. This step is repeated until the preset number of cycles is reached to obtain the final neural network model parameters.

[0071] Example 2

[0072] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the perception prediction method for cross-modal self-supervised learning for embodied intelligent robots as described in Example 1.

[0073] Specifically, the electronic device may include a memory, a processor, and a program stored in the memory, and the processor implements the aforementioned method when executing the program. The device processor includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded from a storage unit to a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored.

[0074] The CPU, ROM, and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus. Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc.

[0075] The communication unit allows the device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks. The processing unit performs the various methods and processes described above, such as the aforementioned steps. For example, in some embodiments, the aforementioned steps can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit.

[0076] In some embodiments, part or all of the computer program may be loaded and / or installed on the device via the ROM and / or the communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps described above may be performed. Alternatively, in other embodiments, the CPU may be configured to perform the aforementioned method in any other appropriate manner (e.g., by means of firmware).

[0077] The functions described above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0078] Example 3

[0079] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the perception prediction method for cross-modal self-supervised learning for embodied intelligent robots as described in Example 1.

[0080] Specifically, the computer-readable storage medium of the present embodiment stores a program, and the aforementioned method is implemented when the program is executed. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or the block diagram to be implemented. The program code can be executed completely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine or completely on a remote machine or server. In the context of the present invention, a computer-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media would include electrical connections based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk-read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0081] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A perception prediction method for cross-modal self-supervised learning of embodied intelligent robots, characterized in that: The motion input vector and the perception input vector of the robot are obtained and used as the input of the cognitive model to obtain the corresponding perception vector and motion vector after reconstruction, wherein the training process of the cognitive model includes the following steps: Step S1, obtaining a motion input vector and a perception input vector related to the robot behavior, and calculating the reconstruction loss corresponding to the motion input vector and the perception input vector by performing self-encoding and decoding respectively; Step S2, calculating a cross-modal mapping loss through cross-modal query based on the motion input vector and the perception input vector after the autoencoder; Step S3, based on the reconstruction loss corresponding to the motion input vector and the perception input vector, and the cross-modal mapping loss, a total loss is weighted, and the parameters of the cognitive model are updated based on the total loss to achieve modeling of implicit intra-modal expression and inter-modal connections based on self-supervised learning.

2. The perception prediction method for cross-modal self-supervised learning of embodied intelligent robots according to claim 1, characterized in that: In the step S1, the motion input vector and the perceptual input vector are self-encoded using a motion encoder and a perceptual encoder respectively, and the self-encoded motion input vector and the perceptual input vector are decoded using a motion decoder and a perceptual decoder respectively, wherein the motion encoder includes a plurality of fully connected layers, the motion decoder includes a plurality of fully connected layers, the perceptual encoder includes a plurality of convolutional layers, and the perceptual decoder includes a plurality of deconvolutional layers.

3. The perception prediction method for cross-modal self-supervised learning of embodied intelligent robots according to claim 1, characterized in that: In step S1, the reconstruction loss L corresponding to the motion input vector and the perception input vector is reconstruct for: Where h(·) represents the encoder, g(·) represents the decoder, and X is the motion input vector or the perceptual input vector.

4. The perception prediction method for cross-modal self-supervised learning of embodied intelligent robots according to claim 1, characterized in that: In step S2, the cross-modal mapping loss L mapping for: Among them, F m 、F p are the motion input vector and the perception input vector after self-encoding, Represents a cross-modal query network.

5. The perception prediction method for cross-modal self-supervised learning of embodied intelligent robots according to claim 1, characterized in that: In step S3, the total loss L is: L=L reconstruct_motion +L reconstruct_perception +β×L mapping Among them, L reconstruct_motion , L reconstruct_perception are the reconstruction losses corresponding to the motion input vector and the perception input vector, respectively, mapping is the cross-modal mapping loss, and β is the weighting coefficient.

6. The perception prediction method for cross-modal self-supervised learning of embodied intelligent robots according to claim 1, characterized in that: In the step S3, based on the total loss, gradient descent back propagation is used to update the parameters of the cognitive model based on Adam optimization to achieve training of the cognitive model.

7. The perception prediction method for cross-modal self-supervised learning of embodied intelligent robots according to claim 1, characterized in that: The motion input vector includes angle and velocity information of robot joints related to robot behavior.

8. The perception prediction method for cross-modal self-supervised learning of embodied intelligent robots according to claim 1, characterized in that: The perception input vector includes contact pressure information on the surface of the robot's mechanical arm and information on the proximity distance to the obstacle related to the robot's behavior.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, wherein the one or more programs include instructions for executing the perception prediction method for cross-modal self-supervised learning for an embodied intelligent robot as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: It includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the perception prediction method for cross-modal self-supervised learning for an embodied intelligent robot as described in any one of claims 1-8.

Images