Multi-modal data feature extraction and optimization method based on teleoperation robot task

Abstract

The invention provides a multi-modal data feature extraction and optimization method based on teleoperation robot tasks, and relates to the technical field of robot imitation learning, and the method comprises the steps: firstly carrying out the data optimization of multi-modal data obtained by a robot through setting the truncation proportion of time steps, and removing meaningless operation data; then, depth information is introduced into the RGB color image in the multi-modal data needing to be processed, and the environment perception capability is enhanced; and finally, performing multi-modal feature mining by applying a DINO technology, wherein the multi-modal feature mining is specifically realized through a pre-training DINOv2 model of a ViT architecture. Through an innovative data optimization and multi-modal feature extraction method, the problems of data redundancy, insufficient feature extraction and low multi-modal information fusion efficiency in imitation learning are solved, and a remarkable actual value is brought to a user.

Description

Technical Field

[0001] The present invention relates to the technical field of robot imitation learning, and particularly to a multi-modal data feature extraction and optimization method based on a teleoperation robot task. Background Art

[0002] With the rapid development of robot technology, the demand for multi-task operations of robots in complex and uncertain environments is increasing. However, when writing task execution rules using traditional manual coding methods, the process is not only cumbersome but also difficult to cope with the dynamic changes in the environment and emergencies. Therefore, imitation learning, as a method of obtaining robot operation skills through expert demonstrations, has gradually become a research hotspot.

[0003] During the process of robot operation learning and task execution, the quality of data and the effectiveness of feature extraction play a crucial role in the success of imitation learning. Although high-quality data collection is the basis of imitation learning, the raw data collected usually contains redundant information or irrelevant parts. If these data are not processed, they may have an adverse impact on the training effect of subsequent models.

[0004] In addition, in complex operation scenarios, robots need to rely on multi-modal information (such as RGB color images, depth images, joint data, etc.) to perceive the environment and complete tasks. As an important data form that can capture the geometric structure of the environment, depth information can significantly enhance the robot's environmental understanding ability. However, traditional feature extraction methods have problems such as insufficient feature extraction and limited feature expression ability when combining multi-modal data.

[0005] In the existing robot imitation learning process, the quality of data and the expression ability of features are crucial for the learning effect of the model and the task execution ability. However, the following problems that urgently need to be solved still exist in the prior art when processing and extracting multi-modal data features:

[0006] (1) The influence of redundant data. During the data collection process, the collected data usually includes operation data after task completion (such as data of the last several time steps). These data do not actually contribute to task completion and may even introduce meaningless noise information. Unfiltered redundant data not only wastes computing resources but may also have a negative impact on the training process of the model, reducing the learning efficiency and generalization ability of the model.

[0007] (2) The difficulty of multi-modal data fusion. Feature fusion of multi-modal data (such as RGB color images and depth images) often faces problems such as information redundancy and mode conflicts. How to fully exploit the complementary characteristics between modalities and generate joint features with strong consistency and high task relevance is a key problem that urgently needs to be solved currently.

[0008] (3) Insufficient feature extraction ability. When traditional feature extraction methods process multi-modal data (such as RGB color images and depth images), they lack the ability of in-depth semantic understanding, and the extracted features often cannot fully represent the key information in the data. Especially in complex task scenarios, existing methods are difficult to extract the fine-grained features that are crucial for the training of imitation learning models.

[0009] Currently, due to the above three problems, the existing 3D model nearest distance query algorithms cannot meet the high real-time requirements of industrial automation scenarios.

[0010] In imitation learning and robot operation tasks, feature extraction and data optimization are the keys to improving model performance and operation efficiency. However, there are still significant limitations in data processing and multi-modal feature extraction in existing technologies, mainly reflected in the following solutions and their deficiencies:

[0011] (1) Directly using raw data for training: In existing imitation learning methods, the collected raw data is usually directly input into the model for training without optimizing the data. Although this can maximize the preservation of data integrity, reduce the data preprocessing link, and save the initial development time, raw data usually contains redundant information, which contributes nothing to task learning and even introduces noise, affecting model performance. In addition, the existence of redundant data leads to waste of computing resources. Especially in large-scale data sets, the training time and storage cost increase significantly. The model is also prone to overfitting when facing irrelevant data, affecting the generalization ability for actual tasks.

[0012] (2) Most traditional methods only use RGB color images as the main source of visual features for feature extraction of image data. In specific simple tasks, RGB color images can provide basic visual information. However, in long-sequence precise operations of robots, relying solely on RGB color images lacks the extraction of depth information about the environmental geometric structure and cannot fully reflect the interaction relationship between the robot and the environment. The feature expression ability of single-modal data is limited in complex task scenarios, especially in tasks involving spatial perception and obstacle avoidance. Moreover, RGB color images are easily affected by light changes and occlusion, resulting in instability of feature extraction.

[0013] (3) Using simple splicing or weighted feature fusion to combine RGB color images and depth images to obtain multimodal information, and enhancing the data representation ability through feature extraction and fusion. Although it can better perceive the spatial layout of the environment, it fails to fully exploit the complementarity between RGB color images and depth images. Moreover, the fused features may contain redundant information or conflicts, affecting the training effect of subsequent models. Traditional methods have insufficient semantic-level understanding of multimodal data and cannot extract deep-level features, resulting in limited feature expression ability. Summary of the Invention

[0014] Aiming at the problems of data redundancy, insufficient multimodal feature extraction, and limited feature fusion ability existing in the prior art, the present invention provides a multimodal data feature extraction and optimization method based on teleoperation robot tasks; by optimizing the data processing flow, deeply mining the complementarity of multimodal information, and introducing advanced self-supervised learning techniques (such as DINO), efficient and high-quality feature extraction and fusion are achieved.

[0015] A multimodal data feature extraction and optimization method based on teleoperation robot tasks includes the following steps:

[0016] Step 1: Optimize the multimodal data obtained by the robot and eliminate meaningless operation data; Step 1.1: Obtain the length of the multimodal data to be processed; Specifically: Obtain the total number of time steps T of each interaction sequence Episode; Step 1.2: Set the truncation ratio of the time steps; Set the truncation ratio a, and consider the truncated part on the right side of the time axis as invalid data. Thus, the length of the valid time steps T' = int((1 - a)×T) is calculated; Step 1.3: Truncate the time step data and filter out invalid data; When reading the multimodal data to be processed, only read the first T' data and eliminate all the data after T'; Step 1.4: Fill and align the dimensions of the truncated data; After the data is truncated, when the truncated data is inconsistent with other data in dimensions, fill the truncated data to align the data dimensions, and identify the filled part with an additional marker; The filling specifically includes setting the remaining time steps to zero or repeating the last valid action.

[0017] Step 2: Introduce depth information into the RGB color images in the multimodal data to be processed to enhance the environmental perception ability; Specifically, by subscribing to the data stream of the depth camera in the ROS robot operating system environment and synchronizing the timestamps with the corresponding RGB color image data stream, a color image and a depth image are obtained from the same perspective at the same moment; the ROS robot operating system compares the minimum timestamps of the multi-camera depth frames and color frames, selects the most matching time series, and discards or delays the processing of the remaining frames; Write the depth map, the joint state of the robot's manipulator, the base motion information, and the sensor data information into the data management file.

[0018] Step 3: Apply the DINO technology for multi-modal feature mining; the DINO technology is specifically implemented through the pre-trained DINOv2 model of the ViT architecture; Input the RGB color image and the depth image into the backbone network of the DINOv2 model respectively: for the depth image, first copy it to the same three color channels as the RGB color image to match the standard input form of the ViT architecture; for the data input of multiple cameras, repeat the same backbone network of the DINOv2 model under the premise of keeping the network weights consistent. DINOv2 first performs patch splitting on the input image, maps each image patch to a high-dimensional feature space, and conducts information interaction and in-depth association on all image patches within the self-attention layer, thereby generating a sequence of feature vector sequences with global context semantics; Remove the [CLS] marker from the output feature vector sequence and rearrange the remaining feature blocks Token to restore the two-dimensional feature map from the one-dimensional vector sequence.

[0019] The beneficial effects produced by adopting the above technical solutions are as follows: The present invention provides a method for feature optimization and extraction based on multi-modal data. Through innovative data optimization and multi-modal feature extraction methods, it solves the problems of data redundancy, insufficient feature extraction, and low multi-modal information fusion efficiency in imitation learning, bringing significant practical value to users, which is mainly reflected in the following aspects: (1) Improve data quality and utilization rate: By eliminating meaningless operation data after task completion, the effectiveness of the data set is significantly improved, and the interference of redundant information on model training is reduced. At the same time, combined with padding and normalization processing, the structural consistency and quality of the data are optimized, ensuring the efficiency and reliability of the model input. This optimization not only reduces the waste of storage and computing resources but also provides a higher-quality data basis for model training; (2) Enhance the robot's environmental perception ability: The present invention introduces depth image information, enabling the robot to more comprehensively perceive the geometric structure and spatial relationships of the environment, compensating for the deficiencies of traditional RGB images in situations such as lighting changes and texture deficiencies. In complex task scenarios, such as object grasping, path planning, and obstacle avoidance, the robot can more accurately understand spatial distances and object positions, significantly improving the success rate and operation accuracy of task execution; (3) Improve the efficiency and performance of model training: Through the introduction of DINOv2, the present invention significantly reduces the dependence on data annotation, enabling users to efficiently train models under conditions of no annotation or few annotations. At the same time, the global information modeling ability of DINOv2 significantly enhances the feature extraction and expression effects of the model, providing stronger generalization ability for imitation learning. This feature enables the robot to more quickly adapt to diverse task scenarios and shorten the model development cycle; (4) Provide broader applicability and scalability: The technical framework of the present invention is applicable to a variety of task scenarios, including but not limited to fields such as robot imitation learning, autonomous driving, industrial automation, and medical assistance. Its versatility and scalability provide flexible solutions for users' application requirements in different scenarios and support the expansion of subsequent functional modules, such as force sensing and real-time feedback control, providing a solid technical foundation for future multi-modal perception and collaborative tasks; (5) Improve the user experience and application effects: Through more accurate and efficient feature extraction and optimization, the present invention significantly improves the user's operation experience and model application effects in actual tasks. The robot can more stably and accurately execute complex tasks in a changing environment, enabling users to obtain higher system reliability and task completion rates, thereby enhancing the practical value and economic benefits of the application. Brief Description of the Drawings

[0020] Figure 1 It is a schematic diagram of the overall process of the feature extraction optimization method in the embodiment of the present invention; Figure 2 It is a flowchart of eliminating meaningless operation data in the embodiment of the present invention; Figure 3 It is a data optimization flowchart in the embodiment of the present invention; Figure 4 It is a schematic diagram of feature extraction in the embodiment of the present invention. Specific Embodiments

[0021] The following combines the drawings and embodiments to further describe in detail the specific embodiments of the present invention. The following embodiments are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0022] A multi-modal data feature extraction and optimization method based on teleoperation robot tasks, as Figure 1As shown, it includes the following steps:

[0023] Step 1: Optimize the multi-modal data obtained by the robot, and eliminate meaningless operation data, such as Figure 3 as shown; In the data collection of imitation learning, the data in the last several time steps after the task is completed usually has no actual contribution to the learning task and may even interfere with the training of the model. Therefore, the present invention optimizes the processing flow of multi-modal data collected by robot training by the following method. The multi-modal data includes RGB color images, depth images, joint data of the robotic arm, speed information, etc., and these data can support embodied training. Eliminating these meaningless time step data can not only improve the effectiveness of the data, but also reduce the interference of redundant information on model training. The processing flow is as Figure 2 shown, and the specific steps include: Step 1.1: Obtain the length of the multi-modal data to be processed; Specifically: Obtain the total number of time steps T of each interaction sequence Episode; Step 1.2: Set the truncation ratio of the time steps; According to the specific requirements of the task and the data analysis results, set the truncation ratio a, and identify the truncated ratio part on the right side of the time axis as invalid data. Thus, the length of the effective time steps T' = int((1 - a)×T) is calculated; in this embodiment, the truncation ratio a = 10% is set, so the effective time step length T' = int(0.9×T); Step 1.3: Truncation of time step data and filtering of invalid data; When reading or processing the multi-modal data, only read the first T' data, such as actions, states (qpos), image frames (RGB color images and depth images), etc., and eliminate all the data after T'; Step 1.4: Filling and dimension alignment of the truncated data; After the data is truncated, when the truncated data is inconsistent with other data (such as data in batch processing) in dimensions, fill the truncated data to align the data dimensions (such as batch processing), and use an additional marker (paddingmask) to identify the filled part; ensure the compatibility of the subsequent processing flow.

[0024] The filling specifically includes setting the remaining time steps to zero or repeating the last valid action; This step effectively solves the problem of interference from redundant data in traditional datasets during training by eliminating meaningless data, filling and aligning, and normalizing, while reducing the requirements for data storage and computing resources. In practical applications, such as robotic arm operation tasks, the optimized data not only simplifies the sequence processing flow but also significantly improves the training efficiency and generalization ability of the model.

[0025] Step 2: Introduce depth information into the RGB color images in the multi-modal data to be processed to enhance the environmental perception ability; To enhance the robot's environmental perception ability in interactive tasks, this solution introduces depth image information based on the original RGB color camera to more accurately capture the shape, distance, and relative position of the scene and target objects in three-dimensional space.

[0026] Specifically, by subscribing to the data stream of the depth camera (such as depth image data) in the ROS robot operating system environment and synchronizing the timestamps with the corresponding RGB color image data stream, color images and depth images are obtained from the same perspective at the same moment. To ensure the effectiveness of the synchronization, the ROS robot operating system compares the minimum timestamps of the multi-camera depth frames and color frames, selects the most matching time series, and discards or delays the processing of the remaining frames; this not only ensures the consistency of depth information, RGB color images, and joint states in the time dimension but also enables the subsequent depth-color fusion algorithm to accurately combine three-dimensional geometric features and texture information.

[0027] Write the depth map, the robotic arm joint state of the robot, the base motion information, and the sensor data information into the data management file; in this embodiment, the HDF5 file is used for data storage and hierarchical management; however, other types of file formats (such as JSON, CSV, etc.) can also be used for storage, and the specific choice depends on the requirements of the application scenario and the data scale. In this embodiment, a hierarchical data management method is adopted, and different data types are classified and stored to facilitate efficient organization and access. This hierarchical management method ensures the structured storage of data and supports fast retrieval and processing.

[0028] In this step, by introducing depth information, the environmental perception ability of the robot in interaction tasks is significantly enhanced. Based on the original RGB color camera and combined with depth image information, the shape, distance, and relative position of the scene and target objects can be captured more accurately. This method subscribes to the data streams of the depth camera and RGB camera in the ROS robot operating system and synchronizes the timestamps, enabling the system to obtain the depth map and color image from the same perspective at the same moment. To ensure data consistency and efficient storage, the depth map and sensor data such as the manipulator joint state and base motion information are stored in a structured file, adopting a hierarchical management method to ensure efficient data organization and fast access. In addition, the ROS robot system ensures the synchronization of the depth map and RGB image through timestamp comparison, ensuring that the subsequent depth-color fusion algorithm can accurately combine three-dimensional geometric features and texture information. This method improves the estimation accuracy of the object space distance and occlusion relationship in complex environments, and enhances the recognition and operation accuracy of the robot. The depth data also provides supplementation for scenes with insufficient lighting or missing textures, enhancing the robustness and adaptability of the system.

[0029] Step 3: Apply the DINO technology for multi-modal feature mining; the DINO technology is specifically implemented through the pre-trained DINOv2 model of the ViT architecture, as Figure 4 shown; To meet the requirements of feature extraction and optimization for multi-modal data (including RGB color images and depth images), the latest DINO (Distillation with No Labels) technology is introduced - especially the DINOv2 version based on the Vision Transformer (ViT) architecture. The core of DINOv2 is to conduct large-scale pre-training in a self-supervised or weakly supervised manner, enabling it to mine the global and local features contained in different modal data under the condition of no annotation or few annotations. By iteratively modeling the image features in the multi-head self-attention layer of the attention mechanism, DINOv2 can simultaneously capture the information of the image at both large scales and fine structures, thus showing high robustness and high transferability when dealing with multi-modal tasks.

[0030] In terms of the specific implementation of multimodal fusion, the RGB color image and the depth image are respectively input into the backbone network of the DINOv2 model: for the depth image, it is first copied to the same three color channels as the RGB color image to match the standard input form of the ViT architecture; for the data input of multiple cameras, the same DINOv2 model backbone network is repeatedly called on the premise of keeping the network weights consistent, making full use of the shared representation ability and global context awareness ability of the model. During the image processing, DINOv2 first performs patch splitting on the input image, maps each image patch to a high-dimensional feature space, and conducts information interaction and in-depth association on all image patches (Patch Tokens) within the self-attention layer, thereby generating a sequence of feature vector sequences with global context semantics; To better integrate into the subsequent task pipeline, the [CLS] token in the output feature vector sequence is removed and the remaining feature blocks Tokens are reshaped, restored from a one-dimensional vector sequence into a two-dimensional feature map similar to the output of a traditional convolutional network; this "sequence-to-feature map" conversion method not only retains the advantage of the Vision Transformer (ViT) architecture in modeling global information at the feature expression level, but also has the convenience of the Convolutional Neural Network (CNN) structure being easily coupled with other spatial operations (such as upsampling, cross-resolution fusion, convolutional modules).

[0031] Through this method, different modality inputs such as RGB color images and depth images complete feature extraction under the same network weights. On the one hand, this method can reduce the additional overhead brought by parallel operation of different networks; on the other hand, it can also ensure that when fusing multimodal features, the feature distributions of each modality are more consistent and alignable.

[0032] The self-supervised pre-training strategy adopted by DINOv2 greatly reduces the dependence on a small amount of annotations when the model extracts multimodal data features; and when processing single-channel depth maps or RGB images with uneven illumination, it can still capture key structural information and depth cues through its self-attention mechanism. Relying on this technology, this solution can efficiently fuse and align key information of different modalities in scenarios with multiple sensors and multiple camera inputs, so that subsequent perception, recognition, detection, or reconstruction modules can obtain more complete and more robust feature expressions, providing a solid foundation for the in-depth mining of multimodal data and multi-task collaboration. Through the above integration and optimization, the features of multimodal data can be fully mined without large-scale manual annotation.

[0033] Through the combination of data optimization, multi-modal feature introduction, and self-supervised learning techniques, the present invention proposes a brand-new solution to the problems of data quality and feature extraction in existing imitation learning. The following are the core innovations of the present invention:

[0034] (1) Efficient optimization of data processing: A refined data truncation strategy is proposed. By removing the last 10% of the time steps in the interaction sequence, meaningless operation data is effectively removed, improving the effectiveness of the data. At the same time, combined with the padding and masking mechanism, the problems of batch processing after truncation and alignment of the model input format are solved. This method significantly reduces data redundancy and improves the efficiency of data storage and calculation.

[0035] (2) Comprehensive integration of multi-modal data: In the processing of imitation learning data, depth information is introduced based on RGB images to construct a multi-modal perception data set. Through precise timestamp alignment and hierarchical storage, the consistency of RGB color images and depth images in the time dimension and spatial information is achieved. This innovation significantly enhances the three-dimensional perception ability of the robot in complex environments and provides a solid foundation for high-precision task execution.

[0036] (3) Feature extraction driven by self-supervised learning: By introducing the DINO technology based on Vision Transformer (ViT), the present invention realizes deep feature extraction of multi-modal data. Compared with traditional methods, the self-supervised feature of DINO enables the model to efficiently extract key features under the condition of no annotation or few annotations and has excellent global information modeling ability. This feature greatly reduces the data annotation cost and improves the robustness and generalization ability of feature extraction.

[0037] The above description is only the preferred embodiments of the present disclosure and the description of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A multi-modal data feature extraction and optimization method based on teleoperation robot tasks, characterized in that It includes the following steps: Step 1: Optimize the multi-modal data obtained by the robot and eliminate meaningless operation data; Step 2: Introduce depth information into the RGB color images in the multi-modal data to be processed to enhance the environmental perception ability; Step 3: Apply the DINO technology for multi-modal feature mining; the DINO technology is specifically implemented through the pre-trained DINOv2 model of the ViT architecture; The specific content of Step 3 is as follows: Input the RGB color image and the depth image into the backbone network of the DINOv2 model respectively: for the depth image, first copy it to the same three color channels as the RGB color image to match the standard input form of the ViT architecture; for the data input of multiple cameras, under the premise of keeping the network weights consistent, repeatedly call the same backbone network of the DINOv2 model. The DINOv2 first performs patch splitting on the input image, maps each image patch to a high-dimensional feature space, and performs information interaction and depth association on all image patches within the self-attention layer, thereby generating a sequence of feature vector sequences with global context semantics; Remove the [CLS] marker in the output feature vector sequence and rearrange the remaining feature patch Tokens to restore the two-dimensional feature map from the one-dimensional vector sequence.

2. A multi-modal data feature extraction and optimization method based on a teleoperation robot task according to claim 1, characterized in that The specific content of Step 1 includes the following steps: Step 1.1: Obtain the length of the multi-modal data to be processed; Specifically: Obtain the total number of time steps T of each interaction sequence Episode; Step 1.2: Set the truncation ratio of the time steps; Set the truncation ratio a, and identify the truncated ratio part on the right side of the time axis as invalid data. Thus, calculate the length of the effective time steps T' = int((1 - a)×T); Step 1.3: Truncate the time step data and filter the invalid data; When reading the multi-modal data to be processed, only read the first T' data and eliminate all the data after T'; Step 1.4: Fill and align the dimensions of the truncated data; After the data is truncated, when the truncated data is inconsistent with other data in dimensions, fill the truncated data to align the data dimensions, and identify the filled part with an additional marker; The specific filling includes setting the remaining time steps to zero or repeating the last valid action.

3. A multimodal data feature extraction and optimization method based on a teleoperated robot task according to claim 1, characterized in that The specific content of Step 2 is as follows: Subscribe to the data stream of the depth camera in the ROS robot operating system environment and synchronize the time stamps with the corresponding RGB color image data stream to obtain the color image and the depth image from the same perspective at the same moment; the ROS robot operating system compares the minimum time stamps of the multi-camera depth frames and color frames, selects the most matching time sequence, and discards or delays the processing of the remaining frames; Write the depth map, the joint state of the robot's manipulator, the base motion information, and the sensor data information into the data management file.

Images