Multi-mode body-equipped intelligent robot control method and device

Abstract

The invention provides a multi-mode intelligent robot control method and device, and the method comprises the steps: obtaining a current mode instruction which at least comprises one of a video instruction, a text instruction, a picture instruction and an audio instruction; on the basis of a pre-trained multi-modal large model, according to the current modal instruction, obtaining post-collapse characterization; on the basis of a pre-trained robot imitation learning network, according to the representation after collapse and current environment observation, predicting an output target action; the multi-mode intelligent robot with the body is controlled to operate according to the target action; wherein the robot imitation learning network is obtained by performing training optimization according to a training sample set formed by single-modal data samples, corresponding environment observation data and real actions of the robot. According to the method, the post-collapse representation is obtained by pre-training the multi-modal large model, and the post-collapse representation is utilized to predict the target action, so that the requirement on multi-modal manual annotation data when a training robot simulates a learning network is avoided in the process, the effect of multi-modal reasoning can be achieved only according to single-modal data training, and the efficiency of multi-modal reasoning is improved. Not only is data collection and labeling cost reduced, but also the reasoning ability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of embodied intelligent robots, and in particular to a multi-modal embodied intelligent robot control method and device. Background Art

[0002] Existing multimodal embodied intelligent robot reasoning relies on massive, paired, one-to-one corresponding multimodal data sample sets for training. However, collecting, labeling and storing these data samples requires a lot of costs, which is a major obstacle to applications in the field of robotics.

[0003] Traditional multimodal learning methods usually process information from different modalities in an independent segmented manner, which makes it difficult to fully capture the interdependencies and semantic integration between different modalities. Therefore, it is still challenging to train a multimodal embodied robot model that can effectively reason in complex real-world scenarios.

[0004] Therefore, it is necessary to solve the problem that the existing multimodal embodied intelligent robot reasoning relies on a large number of training samples and requires high costs. Summary of the invention

[0005] The present invention provides a multimodal embodied intelligent robot control method and device, which are used to overcome the defects that the existing multimodal embodied intelligent robot reasoning relies on a large number of training samples and requires high costs, reduce data collection and annotation costs, and improve the reasoning ability of the model.

[0006] On the one hand, the present invention provides a multimodal embodied intelligent robot control method, comprising: obtaining a current modal instruction, the current modal instruction including at least one of a video instruction, a text instruction, a picture instruction and an audio instruction; based on a pre-trained multimodal large model, obtaining a collapsed representation according to the current modal instruction; based on a pre-trained robot imitation learning network, predicting and outputting a target action according to the collapsed representation and current environmental observations; controlling the multimodal embodied intelligent robot to operate according to the target action; wherein the robot imitation learning network is obtained by training and optimizing a training sample set consisting of a single modal data sample and its corresponding environmental observation data and the robot's real action.

[0007] Furthermore, based on the pre-trained multimodal large model, according to the current modal instruction, a collapsed representation is obtained, including: inputting the current modal instruction into the pre-trained multimodal large model to obtain a connected representation; determining a target modal mean corresponding to the current modal instruction; and performing a difference processing between the connected representation and the target modal mean to obtain the collapsed representation.

[0008] Furthermore, training and optimizing the robot imitation learning network specifically includes: constructing a training sample set, the training sample set including a single modality data sample and its corresponding environmental observation data and the robot's real action; obtaining a training input representation of the robot imitation learning network based on the single modality data sample in the training sample set; using the training input representation and the environmental observation data as model input, using the predicted action as model output, and using the difference between the predicted action and the robot's real action as the training loss, to iteratively optimize the robot imitation learning network.

[0009] Furthermore, based on the single modality data samples in the training sample set, a training input representation of the robot imitation learning network is obtained, including: inputting the single modality data samples into a pre-trained multimodal large model to obtain a connected representation; subtracting the modal mean corresponding to the single modality data sample from the connected representation to obtain a collapsed representation; normalizing the collapsed representation and adding set noise to obtain a destroyed representation, wherein the destroyed representation is the training input representation.

[0010] Furthermore, the set noise includes cosine noise and / or Gaussian noise.

[0011] Furthermore, the modal mean corresponding to the single modal data sample is obtained through the following steps: obtaining a preset number of single modal data samples; extracting sample features of all single modal data samples; summing all sample features according to the feature dimension and dividing by a preset number to obtain the modal mean corresponding to the single modal data sample.

[0012] In the second aspect, the present invention also provides a multimodal embodied intelligent robot control device, including: a current modal instruction acquisition module, used to obtain the current modal instruction, and the current modal instruction includes at least one of a video instruction, a text instruction, a picture instruction and an audio instruction; a collapsed representation acquisition module, used to obtain a collapsed representation according to the current modal instruction based on a pre-trained multimodal large model; a target action prediction module, used to predict and output a target action based on a pre-trained robot imitation learning network, according to the collapsed representation and current environmental observations; a target action operation module, used to control the multimodal embodied intelligent robot to operate according to the target action.

[0013] In a third aspect, the present invention further provides a multimodal embodied intelligent robot, which executes the multimodal embodied intelligent robot control method as described in any one of the above items during operation, or includes the multimodal embodied intelligent robot control device as described above.

[0014] In a fourth aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal embodied intelligent robot control method as described in any one of the above is implemented.

[0015] In a fifth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multimodal embodied intelligent robot control method as described in any one of the above.

[0016] The multimodal embodied intelligent robot control method provided by the present invention obtains the current modal instruction, which includes at least one of a video instruction, a text instruction, a picture instruction, and an audio instruction, and obtains the collapsed representation according to the current modal instruction, and then predicts and outputs the target action based on the pre-trained robot imitation learning network according to the collapsed representation and the current environment observation, thereby controlling the multimodal embodied intelligent robot to operate according to the target action. The method obtains the collapsed representation by pre-training a large multimodal model, and predicts the target action by using the collapsed representation. This process avoids the need for multimodal manually annotated data when training the robot imitation learning network, and can achieve the effect of multimodal reasoning only by training based on a single modal data, which not only reduces the data collection and annotation costs, but also improves the reasoning ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a multi-modal embodied intelligent robot control method provided in an embodiment of the present invention.

[0019] Figure 2 It is a schematic diagram of training optimization of a multimodal large model provided by an embodiment of the present invention.

[0020] Figure 3 It is a training optimization diagram of a robot imitation learning network provided by an embodiment of the present invention.

[0021] Figure 4 It is a schematic diagram of the overall training test of the robot imitation learning network provided by an embodiment of the present invention.

[0022] Figure 5It is a schematic diagram of the overall flow of the multimodal embodied intelligent robot control method provided by an embodiment of the present invention.

[0023] Figure 6 It is a structural schematic diagram of a multimodal embodied intelligent robot control device provided in an embodiment of the present invention.

[0024] Figure 7 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be noted that robots have the ability to understand human multimodal commands (voice commands, text commands, image commands, video commands). At this stage, in order to train multimodal embodied intelligent robots, massive, paired, one-to-one multimodal data sets are required, but such data sets require manual annotation and are costly to collect.

[0027] Taking this into consideration, the present invention proposes a multimodal embodied intelligent robot control method, which avoids the need for multimodal manually labeled data when training a multimodal embodied intelligent robot, and can achieve the effect of multimodal reasoning from a single modality training.

[0028] Figure 1 FIG. 1 is a flow chart showing a multi-modal embodied intelligent robot control method provided by an embodiment of the present invention. Figure 1 As shown, the method includes steps S110-S140, and steps S110-S140 and related steps will be described in detail below.

[0029] S110, obtaining a current modal instruction, where the current modal instruction includes at least one of a video instruction, a text instruction, a picture instruction, and an audio instruction.

[0030] It is easy to understand that in order for the robot to perform an operation, the user first needs to input the current modal instruction. The current modal instruction can be any one of the video instructions, text instructions, picture instructions and audio instructions, or it can be any combination of multiple of the video instructions, text instructions, picture instructions and audio instructions, which can be adjusted according to actual conditions.

[0031] Among them, the video instruction includes video data, the text instruction includes text data, the picture instruction includes image data, and the audio instruction includes audio data.

[0032] For example, in a specific embodiment, the current modal instruction is a video instruction, which includes video data and prompts the robot to operate according to the video data.

[0033] In another specific embodiment, the current modal instruction includes a video instruction and a text instruction, and the video data included in the video instruction and the text data included in the text instruction correspond in content.

[0034] On the basis of obtaining the current modal instruction, further, step S120 is executed.

[0035] S120, obtaining a collapsed representation according to the current modal instruction.

[0036] Specifically, first, useful features can be extracted from the current modal instruction through a pre-trained multimodal large model. Since the multimodal large model is trained and optimized using all open source robot operation data sets of different modalities, a concatenated representation of the output can be obtained by taking the current modal instruction as the input of the multimodal large model. The concatenated representation here not only contains the information of the current modality, but also contains the close connection between the information of the current modality and the information of other modalities corresponding to the current modal instruction.

[0037] The current mode refers to the mode corresponding to the current mode instruction, such as video, text, image, and audio.

[0038] Then, the collapsed representation can be obtained by subtracting the modal mean corresponding to the current mode from the connected representation. The modal mean can be obtained by collecting a large number of data samples of the current mode and calculating the average value according to the feature dimension.

[0039] Subtracting the modal mean corresponding to the current modality from the connected representation is a process also called centralization or de-meaning. The centralized feature representation (i.e., collapsed representation) will be more stable and not affected by the overall data offset. At the same time, it can also reduce the risk of overfitting due to data offset and improve the generalization ability of the model. Of course, the centralized feature representation is also easier to compare and fuse across modalities.

[0040] That is to say, the collapsed representation in this embodiment is a centralized feature representation, which is convenient for subsequent machine learning model processing.

[0041] After obtaining the post-collapse characterization, further, step S130 is performed.

[0042] S130, based on a pre-trained robot imitation learning network, predict and output the target action according to the collapsed representation and the current environmental observation; wherein the robot imitation learning network is obtained by training and optimizing a training sample set consisting of a single modality data sample and its corresponding environmental observation data and the robot's real action.

[0043] A robot imitation learning network is a deep learning model that has been trained with a large amount of example data and is able to learn how to perform specific tasks from examples.

[0044] The current environment observation is the state information of the environment in which the multimodal embodied intelligent robot is currently located, which includes sensor data and other environmental parameters. Among them, sensor data includes image data obtained by the camera, lidar data obtained by the lidar, various force data read by the force sensor, and other environmental parameters include the position, speed, and acceleration of the multimodal embodied intelligent robot.

[0045] After obtaining the collapsed representation and the current environment observation, the collapsed representation and the current environment observation are input into a pre-trained robot imitation learning network to predict and output the next action that the multimodal embodied robot should take, that is, the target action.

[0046] The target action may be a continuous action, such as a change in joint angle, speed adjustment, etc., or a discrete action, such as moving to a certain position, grabbing an object, etc., which is not specifically limited here.

[0047] After determining the target action of the multimodal embodied intelligent robot, further, step S140 is executed.

[0048] S140, controlling the multimodal embodied intelligent robot to operate according to the target action.

[0049] It is easy to understand that the target action is converted into a specific control instruction that the multimodal embodied intelligent robot can directly execute, and then the actuator of the multimodal embodied intelligent robot controls the multimodal embodied intelligent robot to operate according to the converted control instruction.

[0050] It should be noted that the multimodal embodied intelligent robots in this embodiment include but are not limited to household robots, companion robots, factory robots, catering robots and cleaning robots.

[0051] In this embodiment, by obtaining the current modal instruction, the current modal instruction includes at least one of the video instruction, text instruction, picture instruction and audio instruction, and according to the current modal instruction, the collapsed representation is obtained, and then based on the pre-trained robot imitation learning network, the target action is predicted and output according to the collapsed representation and the current environment observation, thereby controlling the multi-modal embodied intelligent robot to operate according to the target action. This method obtains the collapsed representation by pre-training a large multi-modal model, and uses the collapsed representation to predict the target action. This process avoids the need for multi-modal manually annotated data when training the robot imitation learning network. The effect of multi-modal reasoning can be achieved by training only based on a single modal data, which not only reduces the cost of data collection and annotation, but also improves the reasoning ability of the model.

[0052] On the basis of the above-mentioned embodiments, the process of obtaining the post-collapse characterization will be described in detail below.

[0053] Based on the pre-trained multimodal large model, according to the current modal instruction, a collapsed representation is obtained, including: inputting the current modal instruction into the pre-trained multimodal large model to obtain a connected representation; determining the target modal mean corresponding to the current modal instruction; performing a difference processing between the connected representation and the target modal mean to obtain a collapsed representation.

[0054] It can be understood that after obtaining the current modal instruction, the current modal instruction is input into the pre-trained multimodal large model, and after feature extraction, the connected representation can be obtained.

[0055] Among them, the pre-trained multimodal large model is a deep learning model that can process multiple types of data (such as text, images, audio, video, etc.). It is pre-trained on a large-scale dataset to learn cross-modal data representation and association. The pre-trained multimodal large model performs well in a variety of tasks, such as image-text matching, video understanding, cross-modal retrieval, etc.

[0056] Figure 2 FIG. 2 shows a schematic diagram of training optimization of a multimodal large model provided by an embodiment of the present invention. Figure 2 As shown, robot operation datasets of various modalities (including but not limited to text data, image data, video data, and audio data) are used to train and optimize the multimodal large model, so that the pre-trained multimodal large model can understand multiple types of inputs at the same time and learn the complex interaction relationships between different modal data, thereby improving the model's understanding and prediction capabilities.

[0057] The connected representation not only contains the information of the current modality, but also contains the close connection between the information of the current modality and the information of other modalities corresponding to the current modality instructions.

[0058] At the same time, the target modal mean corresponding to the current modal instruction is obtained. Specifically, different modal instructions have corresponding modal means (such as image mean, text mean, video mean and audio mean), and these modal means are pre-calculated and stored. When the current modal instruction is obtained, the current modality corresponding to the current modal instruction can be determined, so that the modal mean corresponding to the current modality, that is, the modal mean corresponding to the current modal instruction, can be directly recorded as the target modal mean.

[0059] Among them, the modal means corresponding to different modal instructions can be obtained by averaging a large number of single modal data samples in the feature dimension.

[0060] Then, the collapsed representation can be obtained by subtracting the target mode mean corresponding to the current mode from the connected representation.

[0061] Subtracting the target modality mean corresponding to the current modality from the connected representation is a process also called centralization or de-meaning. The centralized feature representation (i.e., collapsed representation) will be more stable and not affected by the overall data offset. At the same time, it can also reduce the risk of overfitting due to data offset and improve the generalization ability of the model. Of course, the centralized feature representation is also easier to compare and fuse across modalities.

[0062] After obtaining the collapsed representation, the collapsed representation and the current environment observation are input into a pre-trained robot imitation learning network to obtain the predicted target action.

[0063] Furthermore, the multimodal embodied intelligent robot is controlled to operate according to the target action to achieve the effect of controlling the robot operation.

[0064] In this embodiment, the current modal instruction is input into the pre-trained multimodal large model to obtain the connected representation, and the target modal mean corresponding to the current modal instruction is determined at the same time, and then the connected representation is subtracted from the target modal mean to obtain the collapsed representation, and then based on the pre-trained robot imitation learning network, the target action is predicted and output according to the collapsed representation and the current environmental observation, thereby controlling the multimodal embodied intelligent robot to operate according to the target action. This method obtains the collapsed representation by pre-training the multimodal large model, and uses the collapsed representation to predict the target action. This process avoids the need for multimodal manually annotated data when training the robot imitation learning network. The effect of multimodal reasoning can be achieved by training only on single modal data, which not only reduces the cost of data collection and annotation, but also improves the reasoning ability of the model.

[0065] On the basis of the above embodiments, the training optimization process of the robot imitation learning network will be described in detail below.

[0066] Training and optimizing a robot imitation learning network specifically includes: constructing a training sample set, the training sample set including single modality data samples and their corresponding environmental observation data and real robot actions; obtaining a training input representation of the robot imitation learning network based on the single modality data samples in the training sample set; using the training input representation and environmental observation data as model input, using the predicted action as model output, and using the difference between the predicted action and the real robot action as the training loss, to iteratively optimize the robot imitation learning network.

[0067] First, a training sample set is constructed. In this embodiment, when training and optimizing the robot imitation learning network, multi-modal manual annotation data is not required, but a single-modal data sample is collected.

[0068] The single modality data samples and their corresponding environmental observation data and real robot actions contained in the training sample set may come from actual robot operations or from simulated environments or manual demonstrations, and are not specifically limited here.

[0069] In a specific embodiment, training sample sets are collected from multiple robot platforms, such as the RoboNet project jointly researched by Carnegie Mellon University, University of Pennsylvania and Stanford University, which provides video frame datasets from different robot platforms for learning vision-based robot operation models.

[0070] After constructing the training sample set, the training input representation of the robot imitation learning network is obtained based on the single modal data samples in the training sample set, including: inputting the single modal data samples into the pre-trained multimodal large model to obtain a connected representation; subtracting the modal mean corresponding to the single modal data sample from the connected representation to obtain a collapsed representation; normalizing the collapsed representation and adding set noise to obtain a destroyed representation, which is the training input representation.

[0071] The modal mean corresponding to the single modal data sample is obtained. Specifically, a preset number of single modal data samples are first obtained, and then the sample features of all the single modal data samples are extracted, and then all the sample features are summed according to the feature dimension and divided by the preset number to obtain the modal mean corresponding to the single modal data sample. The preset number can be set according to actual needs and is not specifically limited here.

[0072] The noise may be cosine noise or Gaussian noise, which is not specifically limited here.

[0073] In a specific embodiment, a process of adding cosine noise to the collapsed representation is provided.

[0074] Assume that the original vector (i.e. the collapsed representation) is , whose size is [10,1024]. First, the collapsed representation is normalized to obtain the normalized representation, which can be specifically seen in the following formula (1).

[0075] (1).

[0076] In formula (1), Indicates The normalized result of a vector, that is, the normalized representation.

[0077] Then, an orthogonal random vector is generated for the normalized representation First, generate a random vector , and then subtract its projection on , and then normalize the result. For details, please refer to the following formula (2).

[0078] (2).

[0079] Next, using the given target cosine similarity Generate target vector The destroyed representation is a linear combination of the normalized representation and the orthogonal random vector, as shown in the following formula (3).

[0080] (3).

[0081] The final target vector With the original vector With target cosine similarity .

[0082] The target vector in the above formula (3) is The corrupted representation is a random number generated after the cosine noise size is determined. For example, if the cosine noise is set to 0.7, a floating point number between 0.7 and 1.0 will be randomly sampled each time the cosine noise is added as the target cosine similarity. The numerical value of .

[0083] In another specific embodiment, a process of adding Gaussian noise to the collapsed representation is provided.

[0084] First, the original vector (i.e. the collapsed representation) Normalize it to get the unit vector , please refer to the following formula (4) for details.

[0085] (4).

[0086] Then, we generate Gaussian noise according to the standard deviation of each dimension. Assume Represents the standard deviation of each dimension, the size is

[1024] , then the generated Gaussian noise It can be expressed by the following formula (5).

[0087] (5).

[0088] Next, the generated Gaussian noise is added to the normalized unit vector For details, please refer to the following formula (6).

[0089] (6).

[0090] Finally, the vector after adding noise is normalized to obtain the final vector , which can be seen in the following formula (7).

[0091] (7).

[0092] The vector in formula (7) This is the sign of destruction.

[0093] After obtaining the representation after destruction (training input representation), the training input representation and its corresponding environmental observation data are used as the input of the robot imitation learning network, and the predicted action is output. The difference between the predicted action and the actual action of the robot is used as the training loss, and the robot imitation learning network is iteratively optimized, thereby obtaining a robot imitation learning network that has been trained to convergence.

[0094] also, Figure 3 A schematic diagram of training optimization of a robot imitation learning network provided by an embodiment of the present invention is shown.

[0095] like Figure 3 As shown, first, a single modality data sample (including but not limited to video data, text data, image data, audio data) is input into a multimodal large model (pre-trained multimodal large model) to obtain the output concatenated representation, and then, the modal mean of the modality corresponding to the single modality data sample is subtracted from the concatenated representation to obtain the collapsed representation, and then, the collapsed representation is normalized and noise is added ( Figure 3 The noise in the image is cosine noise, or it can be other noises), and the damaged representation is obtained.

[0096] Finally, the post-destruction representation and environmental observation data are used as the input of the robot imitation learning network for imitation learning. After multiple iterative optimizations, a trained robot imitation learning network can be obtained.

[0097] Figure 4A schematic diagram of the overall training test of the robot imitation learning network provided by an embodiment of the present invention is shown.

[0098] like Figure 4 As shown, when training and optimizing the robot imitation learning network, any single modality data sample can be used as a training sample, without the need for a large number of multimodal data sample annotations. When testing and applying the robot imitation learning network, any single modality data can also be used as a model input to predict the target action. It can be seen that the robot imitation learning network provided in this embodiment can achieve the effect of multimodal data reasoning based on single modality data sample training.

[0099] It should be noted here that using arbitrary single-modal data samples as training samples can avoid the need for multi-modal manually labeled data when training robot imitation learning networks, greatly reducing data collection and labeling costs.

[0100] It should also be noted that during testing and application, a combination of multiple corresponding modal data can also be used as the input of the robot imitation learning network, which is not specifically limited here.

[0101] In this embodiment, by constructing a training sample set, and based on a single modality data sample in the training sample set, the training input representation of the robot imitation learning network is obtained, and then the training input representation and environmental observation data are used as model input, the predicted action is used as model output, and the difference between the predicted action and the robot's real action is used as the training loss, the robot imitation learning network is iteratively optimized, and then based on the pre-trained robot imitation learning network, the target action is predicted and output according to the collapsed representation and the current environmental observation, thereby controlling the multi-modal embodied intelligent robot to operate according to the target action. This method obtains the collapsed representation by pre-training a large multi-modal model, and uses the collapsed representation to predict the target action. This process avoids the need for multi-modal manually annotated data when training the robot imitation learning network. The effect of multi-modal reasoning can be achieved by training only based on single modality data, which not only reduces the cost of data collection and annotation, but also improves the reasoning ability of the model.

[0102] Additionally, in some embodiments, Figure 5 The figure shows the overall process diagram of the multi-modal embodied intelligent robot control method provided by the embodiment of the present invention.

[0103] like Figure 5 As shown, first obtain the current modal instruction, which may be any one or a combination of video instructions, text instructions, picture instructions, and audio instructions.

[0104] Then, the obtained current modal instructions are input into the pre-trained multimodal large model to obtain the post-connection representation.

[0105] Next, the connected representation is subtracted from the target modal mean corresponding to the current modal instruction to obtain the collapsed representation.

[0106] Finally, the collapsed representation and the current environmental observation are used as the input of the robot imitation learning network to obtain the output target action. Then, the multimodal embodied intelligent robot can be controlled to perform according to the target action, thereby achieving precise control of the multimodal embodied intelligent robot.

[0107] Corresponding to the multi-modal embodied intelligent robot control method described in the above embodiments, the present invention also provides a multi-modal embodied intelligent robot control device. Specifically, Figure 6 A schematic diagram of the structure of a multimodal embodied intelligent robot control device provided in an embodiment of the present invention is shown.

[0108] like Figure 6 As shown, the device includes: a current modal instruction acquisition module 610, used to obtain the current modal instruction, and the current modal instruction includes at least one of a video instruction, a text instruction, a picture instruction and an audio instruction; a collapsed representation acquisition module 620, used to obtain a collapsed representation according to the current modal instruction based on a pre-trained multimodal large model; a target action prediction module 630, used to predict and output a target action based on a pre-trained robot imitation learning network, according to the collapsed representation and current environmental observations; a target action operation module 640, used to control a multimodal embodied intelligent robot to operate according to the target action; wherein the robot imitation learning network is obtained by training and optimizing a training sample set consisting of a single modal data sample and its corresponding environmental observation data and the robot's real action.

[0109] In this embodiment, the current modal instruction is obtained by the current modal instruction acquisition module 610, and the current modal instruction includes at least one of a video instruction, a text instruction, a picture instruction, and an audio instruction. The collapsed representation acquisition module 620 acquires the collapsed representation based on the pre-trained multimodal large model and the current modal instruction, and then the target action prediction module 630 predicts and outputs the target action based on the pre-trained robot imitation learning network and the collapsed representation and the current environment observation, so that the target action operation module 640 controls the multimodal embodied intelligent robot to operate according to the target action. The device obtains the collapsed representation through the pre-trained multimodal large model and predicts the target action using the collapsed representation. This process avoids the need for multimodal manually annotated data when training the robot imitation learning network. The effect of multimodal reasoning can be achieved by training only based on single modal data, which not only reduces the cost of data collection and annotation, but also improves the reasoning ability of the model.

[0110] It should be noted that the multimodal embodied intelligent robot control device provided in the embodiment of the present invention and the multimodal embodied intelligent robot control method described in the above embodiments can be referred to each other, and will not be described in detail here.

[0111] In addition, the present invention also provides a multimodal embodied intelligent robot, which includes the multimodal embodied intelligent robot control device described in the above embodiments, and executes the multimodal embodied intelligent robot control method described in the above embodiments during operation. The robot obtains the collapsed representation by pre-training a large multimodal model, and uses the collapsed representation to predict the target action. This process avoids the need for multimodal manually labeled data when training the robot to imitate the learning network. The effect of multimodal reasoning can be achieved by training only on single modal data, which not only reduces the cost of data collection and annotation, but also improves the reasoning ability of the model.

[0112] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the multimodal embodied intelligent robot control method, the method comprising: obtaining the current modal instruction, the current modal instruction at least including one of a video instruction, a text instruction, a picture instruction and an audio instruction; based on the pre-trained multimodal large model, obtaining the collapsed representation according to the current modal instruction; based on the pre-trained robot imitation learning network, predicting and outputting the target action according to the collapsed representation and the current environment observation; controlling the multimodal embodied intelligent robot to operate according to the target action; wherein the robot imitation learning network is obtained by training and optimizing the training sample set consisting of a single modal data sample and its corresponding environment observation data and the robot's real action.

[0113] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0114] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal embodied intelligent robot control method provided by the above-mentioned methods, the method comprising: obtaining a current modal instruction, the current modal instruction comprising at least one of a video instruction, a text instruction, a picture instruction and an audio instruction; based on a pre-trained multimodal large model, obtaining a collapsed representation according to the current modal instruction; based on a pre-trained robot imitation learning network, predicting and outputting a target action according to the collapsed representation and current environmental observations; controlling the multimodal embodied intelligent robot to operate according to the target action; wherein the robot imitation learning network is obtained by training and optimizing a training sample set consisting of a single modal data sample and its corresponding environmental observation data and the robot's real action.

[0115] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal embodied intelligent robot control method, It is characterized in that include: Obtaining a current modal instruction, wherein the current modal instruction includes at least one of a video instruction, a text instruction, a picture instruction, and an audio instruction; Based on the pre-trained multimodal large model, according to the current modal instruction, a collapsed representation is obtained; Based on the pre-trained robot imitation learning network, predict and output the target action according to the collapsed representation and the current environment observation; Controlling the multimodal embodied intelligent robot to perform operations according to the target actions; The robot imitation learning network is obtained by training and optimizing a training sample set consisting of a single modality data sample and its corresponding environmental observation data and the robot's real actions.

2. The multimodal embodied intelligent robot control method according to claim 1, It is characterized in that Based on the pre-trained multimodal large model, according to the current modal instruction, a collapsed representation is obtained, including: Inputting the current modal instruction into a pre-trained multimodal large model to obtain a connected representation; Determining a target modal mean corresponding to the current modal instruction; The post-concatenation representation is subjected to a difference process with the target modal mean to obtain the post-collapse representation.

3. The multimodal embodied intelligent robot control method according to claim 1, It is characterized in that Training and optimizing the robot imitation learning network, including: Constructing a training sample set, wherein the training sample set includes a single modality data sample and its corresponding environmental observation data and real robot movements; Based on the single modality data samples in the training sample set, obtaining a training input representation of the robot imitation learning network; The robot imitation learning network is iteratively optimized with the training input representation and the environment observation data as the model input, the predicted action as the model output, and the difference between the predicted action and the real action of the robot as the training loss.

4. The multimodal embodied intelligent robot control method according to claim 3, It is characterized in that Based on the single modality data sample in the training sample set, a training input representation of the robot imitation learning network is obtained, including: Input the single modality data samples into the pre-trained multimodal large model to obtain the connected representation; Subtracting the modal mean corresponding to the single modal data sample from the concatenated representation to obtain a collapsed representation; The collapsed representation is normalized and set noise is added to obtain a destroyed representation, which is the training input representation.

5. The multimodal embodied intelligent robot control method according to claim 4, It is characterized in that The set noise includes cosine noise and / or Gaussian noise.

6. The multimodal embodied intelligent robot control method according to claim 4, It is characterized in that The modal mean corresponding to the single modal data sample is obtained by the following steps: Obtain a preset number of single modality data samples; Extract sample features of all single modality data samples; All sample features are summed according to the feature dimension and divided by a preset number to obtain the modal mean corresponding to the single modal data sample.

7. A multi-modal embodied intelligent robot control device, It is characterized in that include: A current modal instruction acquisition module, used to acquire a current modal instruction, wherein the current modal instruction includes at least one of a video instruction, a text instruction, a picture instruction, and an audio instruction; A collapsed representation acquisition module, used to acquire the collapsed representation according to the current modal instruction based on the pre-trained multimodal large model; A target action prediction module, used for predicting and outputting a target action based on the collapsed representation and current environment observations based on a pre-trained robot imitation learning network; The target action operation module is used to control the multimodal embodied intelligent robot to operate according to the target action.

8. A multimodal embodied intelligent robot, It is characterized in that During operation, the multimodal embodied intelligent robot control method as described in any one of claims 1 to 6 is executed, or, the multimodal embodied intelligent robot control device as described in claim 7 is included.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the multimodal embodied intelligent robot control method according to any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the multimodal embodied intelligent robot control method as described in any one of claims 1 to 6 is implemented.

Images