Multimodal data feature extraction and optimization method based on teleoperation robot tasks

Abstract

This invention provides a multimodal data feature extraction and optimization method based on teleoperated robot tasks, relating to the field of robot imitation learning technology. This method first optimizes the multimodal data acquired by the robot by setting a time step cutoff ratio to eliminate meaningless operation data. It then introduces depth information into the RGB color images in the multimodal data to be processed to enhance environmental perception. Finally, it applies DINO technology to perform multimodal feature mining, specifically through a pre-trained DINOv2 model based on the ViT architecture. This innovative data optimization and multimodal feature extraction method addresses the problems of data redundancy, insufficient feature extraction, and low multimodal information fusion efficiency in imitation learning, bringing significant practical value to users.

Description

Technical Field

[0001] The present invention relates to the technical field of robot imitation learning, and in particular to a multimodal data feature extraction and optimization method based on a teleoperated robot task. Background Art

[0002] With the rapid development of robotics, the demand for robots to perform multitasking operations in complex and uncertain environments is growing. However, traditional manual coding methods are not only cumbersome when writing task execution rules, but also struggle to cope with dynamic environmental changes and unexpected events. Therefore, imitation learning, a method for acquiring robotic operation skills through expert demonstration, has gradually become a research hotspot.

[0003] In the process of robot manipulation learning and task execution, the quality of data and the effectiveness of feature extraction play a crucial role in the success of imitation learning. Although high-quality data collection is the foundation of imitation learning, the collected raw data often contains redundant information or irrelevant parts. If this data is not processed, it may have a negative impact on the subsequent model training results.

[0004] Furthermore, robots rely on multimodal information (such as RGB color images, depth images, and joint data) to perceive their environment and complete tasks in complex operational scenarios. Depth information, as a crucial data form that captures the geometric structure of the environment, can significantly enhance a robot's environmental understanding. However, traditional feature extraction methods suffer from insufficient feature extraction and limited feature expression capabilities when combining multimodal data.

[0005] In existing robot imitation learning processes, data quality and feature expression are crucial to the model's learning effectiveness and task execution capabilities. However, existing technologies still face the following urgent challenges when processing and extracting multimodal data features:

[0006] (1) The impact of redundant data. During the data collection process, the collected data usually includes operational data after the task is completed (such as the data of the last few time steps). This data does not actually contribute to the completion of the task and may even introduce meaningless noise information. Unfiltered redundant data not only wastes computing resources, but may also have a negative impact on the model training process, reducing the model's learning efficiency and generalization ability.

[0007] (2) Difficulty in multimodal data fusion. Feature fusion of multimodal data (e.g., RGB color images and depth images) often faces problems such as information redundancy and mode conflict. How to fully exploit the complementary characteristics between the modalities and generate joint features with strong consistency and high task relevance is a key issue that needs to be solved urgently.

[0008] (3) Insufficient feature extraction capabilities. Traditional feature extraction methods lack deep semantic understanding capabilities when processing multimodal data (such as RGB color images and depth images), and the extracted features often cannot fully represent the key information in the data. Especially in complex task scenarios, existing methods have difficulty extracting fine-grained features that are crucial for training imitation learning models.

[0009] At present, due to the above three problems, the existing three-dimensional model closest distance query algorithm cannot meet the high real-time requirements of industrial automation scenarios.

[0010] In imitation learning and robotic manipulation tasks, feature extraction and data optimization are key to improving model performance and operational efficiency. However, existing technologies still have significant limitations in data processing and multimodal feature extraction, mainly reflected in the following solutions and their shortcomings:

[0011] (1) Directly using raw data for training: In existing imitation learning methods, the collected raw data is usually directly input into the model for training without optimizing the data. Although this can maximize the preservation of data integrity, reduce the data preprocessing process, and save initial development time, the raw data usually contains redundant information, which does not contribute to task learning and may even introduce noise, affecting model performance. In addition, the presence of redundant data leads to a waste of computing resources, especially on large-scale data sets, which significantly increases training time and storage costs. The model is also prone to overfitting when faced with irrelevant data, affecting its generalization ability for actual tasks.

[0012] (2) Most traditional methods use only RGB color images as the main source of visual features to extract features from image data. In specific simple tasks, RGB color images can provide basic visual information. However, in long-sequence precision operations of robots, relying solely on RGB color images lacks the ability to extract deep information about the geometric structure of the environment and cannot fully reflect the interaction between the robot and the environment. Single-modal data has limited feature expression capabilities in complex task scenarios, especially when it comes to tasks such as spatial perception and obstacle avoidance. Moreover, RGB color images are easily affected by light changes and occlusion, resulting in instability in feature extraction.

[0013] (3) Using simple splicing or weighted feature fusion to combine RGB color images and depth images to obtain multimodal information, and improving data representation capabilities through feature extraction and fusion, although it can better perceive the spatial layout of the environment, it fails to fully tap the complementarity between RGB color images and depth images. In addition, the fused features may contain redundant information or conflicts, affecting the training effect of subsequent models. Traditional methods have insufficient understanding of the semantic level of multimodal data and cannot extract deep features, which limits the feature expression ability. Summary of the Invention

[0014] To address the problems of data redundancy, insufficient multimodal feature extraction, and limited feature fusion capabilities in the existing technology, the present invention provides a multimodal data feature extraction and optimization method based on teleoperated robot tasks. By optimizing the data processing flow, deeply exploring the complementarity of multimodal information, and introducing advanced self-supervised learning techniques (such as DINO), efficient and high-quality feature extraction and fusion are achieved.

[0015] A multimodal data feature extraction and optimization method based on a teleoperation robot task comprises the following steps:

[0016] Step 1: Optimize the multimodal data acquired by the robot and eliminate meaningless operation data;

[0017] Step 1.1: Obtain the length of the multimodal data to be processed;

[0018] Specifically: Get the total time step T of each interaction sequence Episode;

[0019] Step 1.2: Set the time step cutoff ratio;

[0020] Set the truncation ratio a, and identify the truncation ratio part on the right side of the time axis as invalid data. From this, the length of the effective time step T' = int((1-a)×T) is calculated;

[0021] Step 1.3: Truncation of time step data and filtering of invalid data;

[0022] When reading the multimodal data to be processed, only the data before T' is read, and all data after T' is discarded;

[0023] Step 1.4: Padding of truncated data is aligned with the dimensions;

[0024] After the data is truncated, when the truncated data is inconsistent with other data in terms of dimension, the truncated data is padded to align the data dimensions, and the padded part is marked with an additional mark;

[0025] The filling specifically includes setting the remaining time steps to zero or repeating the last valid action.

[0026] Step 2: Introduce depth information into the RGB color image in the multimodal data to enhance environmental perception capabilities;

[0027] Specifically, by subscribing to the depth camera's data stream in the ROS robot operating system environment and synchronizing the timestamps of the corresponding RGB color image data stream, the color image and depth image are acquired from the same perspective at the same time. The ROS robot operating system compares the minimum timestamps of the depth frames and color frames of multiple cameras, selects the time sequence with the best match, and discards or postpones the processing of the remaining frames.

[0028] The depth map, the robot's arm joint status, the base motion information, and the sensor data information are written into the data management file.

[0029] Step 3: Apply DINO technology for multimodal feature mining; the DINO technology is specifically implemented through the pre-trained DINOv2 model of the ViT architecture;

[0030] The RGB color image and depth image are input into the DINOv2 model backbone network separately. For the depth image, it is first copied to the same three color channels as the RGB color image to match the standard input form of the ViT architecture. For data input from multiple cameras, the same DINOv2 model backbone network is repeatedly called while maintaining the consistency of network weights. DINOv2 first performs patch segmentation on the input image, maps each image block to a high-dimensional feature space, and performs information interaction and deep correlation on all image blocks in the self-attention layer, thereby generating a sequence of feature vectors with global contextual semantics.

[0031] Remove the [CLS] tag from the output feature vector sequence and rearrange the remaining feature block tokens to restore the one-dimensional vector sequence to a two-dimensional feature map.

[0032] The beneficial effects of adopting the above technical solution are:

[0033] This paper provides a feature optimization and extraction method based on multimodal data. Through innovative data optimization and multimodal feature extraction methods, it solves the problems of data redundancy, insufficient feature extraction, and low efficiency of multimodal information fusion in imitation learning, bringing significant practical value to users, mainly reflected in the following aspects:

[0034] (1) Improve data quality and utilization: By eliminating meaningless operation data after task completion, the effectiveness of the data set is significantly improved, and the interference of redundant information on model training is reduced. At the same time, combined with padding and normalization processing, the structural consistency and quality of the data are optimized, ensuring the efficiency and reliability of model input. This optimization not only reduces the waste of storage and computing resources, but also provides a higher quality data foundation for model training;

[0035] (2) Enhance the robot's environmental perception ability: This invention introduces depth image information, enabling the robot to more comprehensively perceive the geometric structure and spatial relationship of the environment, making up for the shortcomings of traditional RGB images in situations such as lighting changes and texture loss. In complex task scenarios such as object grasping, path planning, and obstacle avoidance, the robot can more accurately understand spatial distances and object positions, significantly improving the success rate of task execution and operational accuracy;

[0036] (3) Improve model training efficiency and performance: By introducing DINOv2, this invention significantly reduces the reliance on data annotation, enabling users to efficiently train models with no or minimal annotation. At the same time, DINOv2's global information modeling capabilities significantly enhance the model's feature extraction and expression effects, providing stronger generalization capabilities for imitation learning. This feature enables robots to adapt to diverse task scenarios more quickly and shortens the model development cycle;

[0037] (4) Providing wider applicability and scalability: The technical framework of the present invention is applicable to a variety of task scenarios, including but not limited to robot imitation learning, autonomous driving, industrial automation, and medical assistance. Its versatility and scalability provide flexible solutions for users' application needs in different scenarios, and support the expansion of subsequent functional modules, such as force sensing and real-time feedback control, providing a solid technical foundation for future multimodal perception and collaborative tasks;

[0038] (5) Improved user experience and application effectiveness: Through more accurate and efficient feature extraction and optimization, this invention significantly improves the user experience and model application effectiveness in actual tasks. The robot can perform complex tasks more stably and accurately in a changing environment, enabling users to achieve higher system reliability and task completion rates, thereby enhancing the practical value and economic benefits of the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the overall process of the feature extraction optimization method in an embodiment of the present invention;

[0040] Figure 2 This is a data flow chart for eliminating meaningless operations in an embodiment of the present invention;

[0041] Figure 3 This is a data optimization flow chart in an embodiment of the present invention;

[0042] Figure 4 Schematic diagram of feature extraction in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0044] A multimodal data feature extraction and optimization method based on teleoperation robot tasks, such as Figure 1 As shown, the following steps are included:

[0045] Step 1: Optimize the multimodal data acquired by the robot and remove meaningless operation data, such as Figure 3 As shown;

[0046] In the data collection of imitation learning, the data of the last few time steps after the task is completed usually has no actual contribution to the learning task and may even interfere with the training of the model. Therefore, the present invention eliminates time steps in the following way to optimize the processing flow of multimodal data collected for robot training. Multimodal data includes RGB color images, depth images, joint data of the robotic arm, speed information, etc. These data can support embodied training. Eliminating these meaningless time step data can not only improve the validity of the data, but also reduce the interference of redundant information on model training. The processing flow is as follows Figure 2 As shown, the specific steps include:

[0047] Step 1.1: Obtain the length of the multimodal data to be processed;

[0048] Specifically: Get the total time step T of each interaction sequence Episode;

[0049] Step 1.2: Set the time step cutoff ratio;

[0050] According to the specific requirements of the task and the results of data analysis, the truncation ratio a is set, and the truncation ratio portion on the right side of the time axis is considered invalid data. The effective time step length T' is calculated as int((1-a)×T). In this example, the truncation ratio a is set to 10%, so the effective time step length T' is int(0.9×T).

[0051] Step 1.3: Truncation of time step data and filtering of invalid data;

[0052] When reading or processing multimodal data, only read the data before T', such as action (action), state (qpos), image frame (RGB color image and depth image), etc., and discard all data after T';

[0053] Step 1.4: Padding of truncated data is aligned with the dimensions;

[0054] After data is truncated, if the truncated data is dimensionally inconsistent with other data (such as data in batch processing), the truncated data is padded to align the data dimensions (such as batch processing) and the padded part is marked with an additional marker (padding mask) to ensure compatibility with subsequent processing flows.

[0055] The filling specifically includes setting the remaining time steps to zero or repeating the last valid action;

[0056] This step effectively addresses the problem of redundant data in traditional datasets interfering with training by removing meaningless data, performing padding and alignment, and performing normalization. This also reduces the demand for data storage and computing resources. In practical applications, such as robotic arm manipulation tasks, the optimized data not only simplifies the sequence processing workflow but also significantly improves model training efficiency and generalization capabilities.

[0057] Step 2: Introduce depth information into the RGB color image in the multimodal data to enhance environmental perception capabilities;

[0058] To enhance the robot's environmental perception ability in interactive tasks, this solution introduces depth image information based on the original RGB color camera to more accurately capture the shape, distance and relative position of the scene and target objects in three-dimensional space.

[0059] Specifically, by subscribing to the depth camera's data stream (e.g., depth image data) within the ROS robotics operating system environment and synchronizing the timestamps with the corresponding RGB color image data stream, the color and depth images are acquired at the same time and from the same perspective. To ensure the effectiveness of this synchronization, the ROS robotics operating system compares the minimum timestamps of the depth and color frames from multiple cameras, selects the time sequence with the best match, and discards or defers processing of the remaining frames. This not only ensures temporal consistency between depth information, RGB color images, and joint states, but also enables the subsequent depth-color fusion algorithm to accurately combine 3D geometric features with texture information.

[0060] The depth map, along with the robot's arm joint states, base motion information, and sensor data, is written to a data management file. In this embodiment, HDF5 files are used for data storage and hierarchical management. However, other file formats (such as JSON and CSV) can also be used, depending on the application requirements and data size. In this embodiment, a hierarchical data management approach is employed, categorizing and storing different data types for efficient organization and access. This hierarchical management approach ensures structured data storage and supports rapid retrieval and processing.

[0061] By introducing depth information in this step, the robot's environmental perception capabilities during interactive tasks are significantly enhanced. Integrating depth image information with the existing RGB color camera allows for more accurate capture of the shape, distance, and relative position of the scene and target objects. This method subscribes to the data streams from the depth and RGB cameras within the ROS robot operating system and synchronizes timestamps, enabling the system to acquire depth maps and color images from the same perspective at the same time. To ensure data consistency and efficient storage, the depth map, along with sensor data such as the robot arm joint state and base motion information, is stored in structured files, using a hierarchical management approach to ensure efficient data organization and rapid access. Furthermore, the ROS robot system synchronizes the depth map and RGB image through timestamp comparison, ensuring that the subsequent depth-color fusion algorithm can accurately combine 3D geometric features and texture information. This method improves the accuracy of estimating spatial distances and occlusion relationships between objects in complex environments, enhancing the robot's recognition and operation accuracy. Depth data also provides a complementary solution for scenes with insufficient lighting or missing textures, enhancing the system's robustness and adaptability.

[0062] Step 3: Apply DINO technology to perform multimodal feature mining; the DINO technology is specifically implemented through the pre-trained DINOv2 model of the ViT architecture, such as Figure 4 As shown;

[0063] To address the need for feature extraction and optimization of multimodal data (including RGB color images and depth images), we have introduced the latest DINO (Distillation with No Labels) technology—specifically, the DINOv2 version based on the Vision Transformer (ViT) architecture. The core of DINOv2 lies in large-scale pre-training using self-supervision or weak supervision, enabling it to mine global and local features contained in data from different modalities with little or no annotation. By iteratively modeling image features within the multi-head self-attention layer of the attention mechanism, DINOv2 can simultaneously capture both large-scale and fine-scale image information, demonstrating high robustness and transferability when handling multimodal tasks.

[0064] In the specific implementation of multimodal fusion, the RGB color image and depth image are input into the DINOv2 model backbone network separately. For the depth image, it is first copied to the same three color channels as the RGB color image to match the standard input form of the ViT architecture. For data input from multiple cameras, the same DINOv2 model backbone network is repeatedly called while maintaining consistent network weights, fully leveraging the model's shared representation capabilities and global context perception capabilities. During image processing, DINOv2 first patches the input image, mapping each image block to a high-dimensional feature space. Within the self-attention layer, all image blocks (patch tokens) are interacted and deeply associated, generating a sequence of feature vectors with global contextual semantics.

[0065] To better integrate into subsequent task pipelines, the [CLS] tag in the output feature vector sequence is removed and the remaining feature block tokens are reshaped, restoring the one-dimensional vector sequence into a two-dimensional feature map similar to the output of a traditional convolutional network. This "sequence-to-feature map" conversion method retains the advantage of the Vision Transformer (ViT) architecture in modeling global information at the feature expression level, while also taking advantage of the convolutional neural network (CNN) structure's ease of coupling with other spatial operations (such as upsampling, cross-resolution fusion, and convolution modules).

[0066] This method allows feature extraction from different modal inputs, such as RGB color images and depth images, to be performed using the same network weights. This reduces the overhead associated with running different networks in parallel and ensures that the feature distributions of each modality are more consistent and aligned when multimodal features are fused.

[0067] DINOv2's self-supervised pre-training strategy significantly reduces the model's reliance on minimal annotations when extracting features from multimodal data. Furthermore, its self-attention mechanism can still capture key structural information and depth cues when processing single-channel depth maps or RGB images with uneven illumination. Leveraging this technology, this solution can efficiently fuse and align key information from different modalities in scenarios with multi-sensor and multi-camera inputs, enabling subsequent perception, recognition, detection, or reconstruction modules to obtain more complete and robust feature representations, providing a solid foundation for deep mining of multimodal data and multi-task collaboration. Through this integration and optimization, multimodal data features can be fully mined without the need for large-scale manual annotation.

[0068] This invention combines data optimization, multimodal feature introduction, and self-supervised learning techniques to propose a new solution to the data quality and feature extraction issues in existing imitation learning. The following are the core innovations of this invention:

[0069] (1) Efficient optimization of data processing: A refined data truncation strategy is proposed. By eliminating the last 10% of time steps in the interaction sequence, meaningless operation data is effectively removed, thereby improving data validity. At the same time, combined with padding and masking mechanisms, the problem of batch processing and model input format alignment after truncation is solved. This method significantly reduces data redundancy and improves the efficiency of data storage and computing.

[0070] (2) Comprehensive integration of multimodal data: In imitation learning data processing, depth information is introduced into RGB images to construct a multimodal perception dataset. Through precise timestamp alignment and hierarchical storage, consistency between RGB color images and depth images in terms of temporal and spatial information is achieved. This innovation significantly enhances the robot's three-dimensional perception capabilities in complex environments and provides a solid foundation for high-precision task execution.

[0071] (3) Feature extraction driven by self-supervised learning: By introducing the DINO technology based on the Vision Transformer (ViT), this paper achieves deep feature extraction from multimodal data. Compared with traditional methods, the self-supervised nature of DINO enables the model to efficiently extract key features with no or minimal annotations and has excellent global information modeling capabilities. This feature greatly reduces the cost of data annotation while improving the robustness and generalization of feature extraction.

[0072] The above description is merely an illustration of the preferred embodiments of the present disclosure and the technical principles employed. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A multimodal data feature extraction and optimization method based on teleoperation robot tasks, characterized in that: The following steps are involved: Step 1: Optimize the multimodal data acquired by the robot and eliminate meaningless operation data; Step 2: Introduce depth information into the RGB color image in the multimodal data to enhance environmental perception capabilities; Step 3: Apply DINO technology for multimodal feature mining; the DINO technology is specifically implemented through the pre-trained DINOv2 model of the ViT architecture; Step 3 specifically includes: inputting the RGB color image and depth image into the DINOv2 model backbone network respectively: for the depth image, first copying it to the same three color channels as the RGB color image to match the standard input form of the ViT architecture; for data input from multiple cameras, repeatedly calling the same DINOv2 model backbone network while maintaining consistent network weights. DINOv2 first performs patch segmentation on the input image, maps each image block to a high-dimensional feature space, and performs information interaction and deep correlation on all image blocks in the self-attention layer, thereby generating a sequence of feature vectors with global contextual semantics; Remove the [CLS] tag from the output feature vector sequence and rearrange the remaining feature block tokens to restore the one-dimensional vector sequence to a two-dimensional feature map.

2. A multimodal data feature extraction and optimization method based on a teleoperation robot task according to claim 1, characterized in that: The step 1 specifically includes the following steps: Step 1.1: Obtain the length of the multimodal data to be processed; Specifically: Get the total time step T of each interaction sequence Episode; Step 1.2: Set the time step cutoff ratio; Set the truncation ratio a, and identify the truncation ratio part on the right side of the time axis as invalid data. From this, the length of the effective time step T' = int((1-a)×T) is calculated; Step 1.3: Truncation of time step data and filtering of invalid data; When reading the multimodal data to be processed, only the data before T' is read, and all data after T' is discarded; Step 1.4: Padding of truncated data is aligned with the dimensions; After the data is truncated, when the truncated data is inconsistent with other data in terms of dimension, the truncated data is padded to align the data dimensions, and the padded part is marked with an additional mark; The filling specifically includes setting the remaining time steps to zero or repeating the last valid action.

3. The multimodal data feature extraction and optimization method based on teleoperation robot tasks according to claim 1 is characterized in that: Step 2 specifically includes: acquiring a color image and a depth image from the same perspective at the same time by subscribing to the depth camera's data stream and synchronizing the timestamps of the corresponding RGB color image data stream in the ROS robot operating system environment; comparing the minimum timestamps of the depth frames and color frames of multiple cameras, selecting the best matching time sequence, and discarding or postponing the processing of the remaining frames; Write the depth map, the robot's arm joint status, base motion information, and sensor data information into the data management file.

Images