Multimodal data augmentation system for robots in industrial close-range grasping scenarios

Abstract

This invention provides a multimodal data augmentation system for robots in industrial near-field grasping scenarios, relating to the field of industrial robot technology. The system includes: a data preprocessing module, which acquires an RGB image, a depth image, and trajectory data composed of robot execution trajectory and visual information; a background reconstruction module, which reconstructs the background of the RGB image; an adaptive lighting enhancement module, which generates multiple versions of lighting perturbation images from the reconstructed RGB image and fuses them to obtain an enhanced RGB image; a depth information reconstruction module, which reconstructs and completes missing or invalid regions in the original depth image; a multimodal data synchronization verification module, which performs consistency verification; and an enhanced data output module, which repackages and outputs the enhanced RGB image, depth image, trajectory data, and corresponding checksums. This system significantly improves the quality and robustness of multimodal data without requiring replacement of existing hardware.

Description

Technical Field

[0001] This invention relates to the field of industrial robot technology, and in particular to a multimodal data augmentation system for robots designed for close-range grasping scenarios in industry. Background Technology

[0002] Currently, multimodal perception systems for close-range grasping in industrial robots primarily employ traditional depth cameras in conjunction with RGB cameras to acquire spatial and visual information of the target area. In practical applications, to address the issue of depth cameras failing to focus effectively and resulting in missing depth information when the grasping distance is too close (e.g., less than 30cm), existing technologies typically compensate using the following two methods:

[0003] (1) Repeated data collection method:

[0004] By repeatedly collecting RGB and depth data of the same target using a mobile grasping device, the aim is to obtain a relatively complete dataset at different distances and angles. However, this method requires high acquisition and annotation costs, is inefficient, and is difficult to adapt to complex and dynamic scenes.

[0005] (2) Method of replacing with higher-specification hardware:

[0006] This approach directly selects high-end industrial-grade depth cameras or multi-type fusion sensors that can support smaller working distances or stronger sensing capabilities. While this solution alleviates the problem of missing close-range data to some extent, it significantly increases equipment costs and system integration complexity, and does not fundamentally address issues such as different environments and diverse data requirements.

[0007] In addition, existing technologies for multimodal data augmentation often only employ standard data preprocessing procedures, lacking systematic methods for optimization and enhancement specifically for non-ideal industrial environments (such as lighting and background). Data consistency verification methods are also relatively limited. Therefore, these existing technologies struggle to efficiently and cost-effectively achieve high-quality and diverse multimodal data acquisition and augmentation in complex, close-range industrial scenarios. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a robot multimodal data enhancement system for industrial close-range grasping scenarios, thereby achieving intelligent enhancement of multimodal data.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a robot multimodal data augmentation system for industrial close-range grasping scenarios, including a data preprocessing module, a background reconstruction module, an adaptive illumination enhancement module, a depth information reconstruction module, a multimodal data synchronization verification module, and an enhanced data output module;

[0010] The data preprocessing module acquires an HDF5 data format file composed of the robot's execution trajectory and visual information, and performs preprocessing on it; the HDF5 data format file includes RGB images, depth images, and trajectory data;

[0011] The background reconstruction module performs background reconstruction on RGB images to enhance image diversity;

[0012] The adaptive illumination enhancement module takes the RGB image after background reconstruction output by the background reconstruction module as input, generates multiple versions of illumination perturbation images within the parameter range of brightness, contrast and gamma value, and performs fusion processing on the multiple versions of illumination perturbation images based on a preset fusion strategy to obtain an enhanced RGB image that is robust under various illumination conditions.

[0013] The depth information reconstruction module reconstructs and completes missing or invalid regions in the original depth image.

[0014] The multimodal data synchronization and verification module is used to synchronously verify the enhanced RGB image, the reconstructed and completed depth image and trajectory data, and generate corresponding verification codes to ensure the consistency of the RGB image, depth image and trajectory data during the enhancement process.

[0015] The enhanced data output module repackages and outputs the enhanced RGB image, depth image, trajectory data, and corresponding checksum to form a dataset that can be directly used for training or deployment.

[0016] Furthermore, the data preprocessing module completes data preprocessing through the following steps:

[0017] Perform integrity checks and outlier detection on HDF5 data format files;

[0018] Extract RGB and depth image data from HDF5 data format files and associate trajectory data based on timestamp alignment technology;

[0019] Generate hash check codes for each frame of data in the RGB and depth images for consistency verification.

[0020] Furthermore, the background reconstruction module includes a SAM segmentation engine, a Gaussian background generator, and a scene reconstructor;

[0021] The SAM segmentation engine uses the SAM algorithm to automatically extract the foreground region of the RGB image, including the robotic arm and the target object.

[0022] The Gaussian background generator applies Gaussian blur to the background area after RGB image segmentation to construct a uniform and controllable blurred background.

[0023] The scene reconstructor seamlessly stitches together the blurred background and foreground areas to form a natural and realistic enhanced RGB image.

[0024] Furthermore, the SAM segmentation engine is based on the SAM2 algorithm and processes the input RGB image. The foreground region is extracted, and the processing flow is as follows:

[0025] First, the RGB image is processed by an image encoder to obtain features:

[0026] ;

[0027] in, This represents an image feature extraction network. Parameters for the image feature extraction network; This represents the encoded image features.

[0028] Then, a location embedding term is added to the encoded image features:

[0029] ;

[0030] in, To add location embedding terms to the image features, This is the location embedding matrix, used to add spatial location information to the encoded feature map;

[0031] Then, the image features after adding the position embedding term are fused with the cue information through an attention mechanism, and the mask probability map is obtained through a mask decoding function:

[0032] ;

[0033] ;

[0034] in, For the prompt encoding function, To provide the parameters for the encoding function; This indicates a prompt message for the input; For mask decoding function, These are the parameters for mask decoding; The output is a mask probability map, representing the probability that a pixel belongs to the foreground. , These are mask probability maps. Height and width;

[0035] Finally, the foreground mask is obtained through thresholding:

[0036] ;

[0037] in, For threshold parameters; when When, it represents a pixel. It belongs to the foreground area; otherwise, it belongs to the background area.

[0038] Furthermore, the adaptive illumination enhancement module introduces a random seed control mechanism to perturb the brightness, contrast, and gamma value of the RGB image n times within a range of 10%, generating RGB images with different perturbation versions.

[0039] Furthermore, the specific method by which the adaptive illumination enhancement module perturbs the brightness, contrast, and gamma value of the RGB image is as follows:

[0040] A random seed value s is set, and a perturbation coefficient in the range of [-0.1, +0.1] is generated based on a pseudo-random number generator to perturb the brightness, contrast, and gamma values ​​of the RGB image; the calculation method is as follows:

[0041] ;

[0042] ;

[0043] ;

[0044] ;

[0045] in, This is the image after brightness and contrast perturbation of the input RGB image. To perturb the brightness and contrast of an image by adding a gamma value, This is the contrast adjustment factor. This is the brightness offset. Gamma correction factor; , , Representing contrast, brightness, and gamma perturbation factors respectively, following an interval... Uniform distribution within ,in A random seed is used to control the repeatability of the perturbation;

[0046] An enhanced RGB image is formed by fusing RGB images with different levels of perturbation using a hybrid strategy executor. The fusion formula is as follows:

[0047] ;

[0048] in, This represents the pixel value at pixel (x,y) in the RGB image obtained after the i-th perturbation. This represents the weight of the RGB image obtained after the i-th perturbation, which is adaptively assigned based on the quality metric. This is the enhanced RGB image obtained after fusing the RGB images obtained after n perturbations.

[0049] Furthermore, during the system training phase, the depth information reconstruction module employs a depth completion pre-trained network model, DenseDepth, based on an encoder-decoder structure, to perform depth prediction on the RGB-D image fused from the RGB image and the depth image. This enables the completion and fusion of missing depth regions in the depth image, thereby improving the integrity and depth consistency of the training data.

[0050] First, the RGB image and the depth image are fused to obtain the RGB-D image. , Given the height and width of an RGB-D image, the pre-trained network model DenseDepth is used to predict the depth of the input RGB-D image, resulting in a predicted depth map. ;

[0051] Then, predict the depth map Compared with the original depth map In the missing area The internal fusion and replacement are performed to obtain the completed depth map. ;

[0052] During training, the DenseDepth network is updated end-to-end by minimizing a multinomial joint loss function:

[0053] ;

[0054] in, The scale-invariant logarithmic error term, For gradient smoothing term, For structural similarity terms, This is the anti-Huber error term; , , , The weights for each loss item.

[0055] Furthermore, the multimodal data synchronization verification module includes a timing alignment checker and an integrity verifier;

[0056] The temporal alignment checker ensures that multimodal data such as RGB images, depth images, and trajectory data are aligned at the timestamp level;

[0057] The integrity verifier compares the initial hash checksums of the original RGB image and depth image with the multimodal data of the enhanced RGB image and the reconstructed and completed depth image to verify the consistency of the content.

[0058] The beneficial effects of adopting the above technical solution are as follows: The intelligent enhancement system for multimodal data of robots for industrial close-range grasping scenarios provided by this invention aims to solve key problems existing in the prior art, such as lack of depth information, insufficient data diversity, difficulty in ensuring consistency, and low system integration. Unlike existing methods that rely on high-end hardware or repetitive data collection, this invention adopts a systematic approach with modular definitions. Through modular architecture and multi-technology integration, it significantly improves the quality and robustness of multimodal data without replacing existing hardware. Attached Figure Description

[0059] Figure 1 This is a structural block diagram of a robot multimodal data intelligent enhancement system for industrial close-range grasping scenarios provided in Embodiment 1 of the present invention. Detailed Implementation

[0060] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0061] Currently, multimodal perception systems for close-range grasping in industrial robots have the following drawbacks:

[0062] (1) Lack of depth information and inability to effectively compensate: Existing depth cameras generally have the problem of focusing or ranging failure when they are too close to the target object (e.g., less than 30cm), resulting in the loss of key depth information. Traditional multiple acquisition methods cannot fundamentally solve this problem, and can only partially increase the amount of data, but cannot guarantee completeness and accuracy; while replacing high-end hardware is costly and has integration and compatibility risks, and cannot meet the actual industrial needs of low cost and easy deployment.

[0063] (2) Insufficient data diversity and robustness: The current reliance on fixed acquisition conditions (such as the same lighting environment and the same background) easily leads to a single distribution of dataset samples, resulting in insufficient model generalization ability and robustness. Most existing enhancement methods are general algorithms, lacking specific optimization for the characteristics of industrial scenarios (such as reflection, occlusion, and complex workpiece backgrounds), and cannot meet the high reliability grasping requirements in dynamic and complex environments.

[0064] (3) Lack of guarantee mechanism for data consistency and integrity: During the acquisition and enhancement of multimodal data, existing solutions rarely perform automatic consistency or integrity checks on trajectory data and image data, which can easily lead to data mismatch, loss or inconsistency, thereby affecting model training and actual operation safety.

[0065] (4) Low system integration and complex operation and maintenance: Existing solutions are mostly single-point technology or pure hardware upgrades, lacking a systematic design from data collection, enhancement, fusion to consistency verification, resulting in the overall process being highly dependent on manual intervention, insufficient system integration and intelligence level, and high maintenance costs.

[0066] To address the aforementioned shortcomings, this invention proposes a multimodal data augmentation system for robots designed for close-range grasping scenarios in industry, achieving the following:

[0067] By employing methods such as adaptive background fusion and random seed multi-parameter control, we can effectively compensate for the lack of depth information and improve data diversity and robustness.

[0068] Mechanisms such as trajectory-image integrated hash verification are adopted to achieve automated consistency and integrity checks of multimodal data during processing;

[0069] By replacing high-cost hardware upgrades with modular integration, the application threshold and deployment costs are significantly reduced, and the stability and reliability of industrial robots in intelligent grasping in complex close-range scenarios are improved.

[0070] Example 1:

[0071] In this embodiment, a robot multimodal data augmentation system for industrial close-range grasping scenarios, such as... Figure 1 As shown, it includes a data preprocessing module, a background reconstruction module, an adaptive lighting enhancement module, a depth information reconstruction module, a multimodal data synchronization and verification module, and an enhanced data output module;

[0072] The data preprocessing module acquires an HDF5 data format file composed of the robot's execution trajectory and visual information, and performs preprocessing on it; the HDF5 data format file includes RGB images, depth images, and trajectory data;

[0073] The background reconstruction module performs background reconstruction on RGB images to enhance image diversity;

[0074] The adaptive illumination enhancement module takes the RGB image after background reconstruction output by the background reconstruction module as input, generates multiple versions of illumination perturbation images within the parameter range of brightness, contrast, and gamma value, and performs fusion processing on the multiple versions of illumination perturbation images based on a preset fusion strategy (including weighted average, region stitching, or quality score filtering, etc.) to obtain an enhanced RGB image that is robust under various illumination conditions.

[0075] The depth information reconstruction module reconstructs and completes missing or invalid regions in the original depth image.

[0076] The multimodal data synchronization and verification module is used to synchronously verify the enhanced RGB image, the reconstructed and completed depth image and trajectory data, and generate corresponding verification codes to ensure the consistency of the RGB image, depth image and trajectory data during the enhancement process.

[0077] The enhanced data output module will repackage and output the enhanced RGB images, depth images, trajectory data and corresponding check codes from the above modules to form a dataset that can be directly used for training or deployment.

[0078] In this embodiment, the data preprocessing module completes data preprocessing through the following steps:

[0079] Perform integrity checks and outlier detection on HDF5 data format files;

[0080] Extract RGB and depth image data from HDF5 data format files and associate trajectory data based on timestamp alignment technology;

[0081] Generate a hash checksum for each frame of data in the RGB and depth images for consistency verification in subsequent processes.

[0082] The data preprocessing module effectively ensures the basic quality of the original multimodal data stream and provides reliable input for subsequent enhancement processes;

[0083] In this embodiment, the background reconstruction module includes a SAM segmentation engine, a Gaussian background generator, and a scene reconstructor; wherein, the SAM segmentation engine adopts the Segment Anything Model (SAM) algorithm to automatically extract the foreground region including the robotic arm and the target object in the RGB image;

[0084] The Gaussian background generator applies Gaussian blur to the background area after RGB image segmentation to construct a uniform and controllable blurred background.

[0085] The scene reconstructor seamlessly stitches together the blurred background and foreground areas to create a natural and realistic enhanced image;

[0086] This background reconstruction module primarily enhances data diversity, especially in complex industrial contexts, improving the ability to identify target areas.

[0087] In this embodiment, the SAM segmentation engine is based on the Segment Anything Model (SAM2) algorithm to segment the input RGB image. The foreground region is extracted, and the processing flow is as follows:

[0088] First, the RGB image is processed by an image encoder to obtain features:

[0089] ;

[0090] in, This represents an image feature extraction network. Parameters for the image feature extraction network; This represents the encoded image features.

[0091] Then, a location embedding term is added to the encoded image features:

[0092] ;

[0093] in, To add location embedding terms to the image features, This is the location embedding matrix, used to add spatial location information to the encoded feature map;

[0094] Then, the image features after adding the position embedding term are fused with the cue information through an attention mechanism, and the mask probability map is obtained through a mask decoding function:

[0095] ;

[0096] ;

[0097] in, For the prompt encoding function, To provide the parameters for the encoding function; Indicates input prompts (such as dots, boxes, or thick masks); For mask decoding function, These are the parameters for mask decoding; The output is a mask probability map, representing the probability that a pixel belongs to the foreground. , These are mask probability maps. Height and width;

[0098] Finally, the foreground mask is obtained through thresholding:

[0099] ;

[0100] in, For threshold parameters; when When, it represents a pixel. It belongs to the foreground area; otherwise, it belongs to the background area.

[0101] This enables automatic segmentation of the foreground and background regions, including the robotic arm and the target object.

[0102] In this embodiment, the adaptive illumination enhancement module introduces a random seed control mechanism to perturb the brightness, contrast, and gamma value of the RGB image n times within a range of 10%, generating RGB images with different perturbation versions.

[0103] The specific method for perturbing the brightness, contrast, and gamma values ​​of an RGB image is as follows:

[0104] A random seed value s is set, and a perturbation coefficient in the range of [-0.1, +0.1] is generated based on a pseudo-random number generator to perturb the brightness, contrast, and gamma values ​​of the RGB image; the calculation method is as follows:

[0105] ; ;

[0106] ;

[0107] ;

[0108] in, This is the image after brightness and contrast perturbation of the input RGB image. To perturb the brightness and contrast of an image by adding a gamma value, This is the contrast adjustment factor. This is the brightness offset. Gamma correction factor; , , Representing contrast, brightness, and gamma perturbation factors respectively, following an interval... Uniform distribution within ,in A random seed is used to control the repeatability of the perturbation. By perturbing the above parameters, the image brightness, contrast, and gamma value are randomly adjusted, thereby enhancing the robustness of the model in multi-lighting environments.

[0109] The RGB images processed with different levels of perturbation are then fused using a hybrid policy executor to form a more robust enhanced RGB image. The fusion formula is as follows:

[0110] ;

[0111] in, This represents the pixel value at pixel (x,y) in the RGB image obtained after the i-th perturbation. The weights represent the weights of the RGB image obtained after the i-th perturbation, which are adaptively assigned based on quality metrics such as sharpness score, structural similarity (SSIM), or information entropy. The enhanced RGB image is obtained by fusing RGB images after n perturbations. This enhanced RGB image maintains high robustness under various lighting conditions.

[0112] This adaptive illumination enhancement module solves the problem of insufficient data stability in existing acquisition systems under different lighting conditions, and improves the model's ability to adapt to changes in illumination.

[0113] In this embodiment, the depth information reconstruction module uses the DenseDepth pre-trained network model based on the encoder-decoder structure to predict the depth of the RGB-D image fused from the RGB image and the depth image during the system training phase, so as to complete and fuse the missing depth regions of the depth image, thereby improving the integrity and depth consistency of the training data.

[0114] First, the RGB image and the depth image are fused to obtain the RGB-D image. , Given the height and width of an RGB-D image, the pre-trained network model DenseDepth is used to predict the depth of the input RGB-D image:

[0115] ;

[0116] in, This indicates a pre-trained DenseDepth network whose encoder uses a DenseNet or ResNet architecture. These are the parameters of the pre-trained DenseDepth network. To predict the depth map;

[0117] Then, predict the depth map Compared with the original depth map In the missing area The internal fusion and replacement are performed to obtain the completed depth map. :

[0118] ;

[0119] The above completed depth map As an augmentation input to the training samples, it is used to optimize the overall multimodal data of the system, thereby enhancing the parameters of the DenseDepth network.

[0120] During training, the DenseDepth network is updated end-to-end by minimizing a multinomial joint loss function:

[0121] ;

[0122] in, The scale-invariant logarithmic error term, For gradient smoothing term, For structural similarity terms, This is the anti-Huber error term; , , , The weights for each loss item.

[0123] This training strategy enables the system to automatically complete deeply missing regions and optimize data consistency during the training phase, significantly improving the integrity and training quality of the multimodal augmented dataset.

[0124] This depth information reconstruction module solves the problem of unavailable depth information during close-range grasping.

[0125] In this embodiment, the multimodal data synchronization verification module includes a temporal alignment checker and an integrity verifier. The temporal alignment checker ensures that the multimodal data, such as RGB images, depth images, and trajectory data, are aligned at the timestamp level. The integrity verifier compares the initial hash checksums of the original RGB images and depth images with the multimodal data, such as the enhanced RGB images and the reconstructed and completed depth images, to verify the consistency of the content. This verification mechanism can prevent data mismatch or corruption, thereby ensuring the availability and engineering reliability of the enhanced dataset.

[0126] Example 2:

[0127] In this embodiment, based on Embodiment 1, the robot multimodal data augmentation system for industrial close-range grasping scenarios also includes a data preprocessing module, a background reconstruction module, an adaptive lighting enhancement module, a depth information reconstruction module, a multimodal data synchronization verification module, and an enhanced data output module.

[0128] The data preprocessing module can also acquire encapsulated data in formats such as ROS Bag, JSON+binary, and TFRecord.

[0129] The background reconstruction module can also perform background reconstruction on RGB images using the following three background processing methods:

[0130] Use an image generation model (such as Stable Diffusion) to generate the background;

[0131] Randomly replace the background image using a background material database;

[0132] The background is reconstructed from 3D point clouds, rendered as an image, and then blended.

[0133] Image style transfer or salient region masking can be used as alternatives for stitching.

[0134] The adaptive lighting enhancement module can also use the following methods to enhance lighting:

[0135] The Retinex algorithm is used to model global and local lighting to achieve lighting enhancement;

[0136] Use HDR fusion of multi-exposure images for lighting enhancement;

[0137] Illumination enhancement is achieved by simulating illumination perturbations using GAN models (such as IlluminationGAN).

[0138] Lighting enhancement is achieved by perturbation of the HSV color space or by simulating local shadows.

[0139] The depth information reconstruction module can also perform depth filling on depth images using the following methods:

[0140] Depth is reconstructed using a binocular camera and stereo matching algorithms (such as StereoSGBM and PSMNet);

[0141] Deep inference is performed using an RGB-D joint optimization network;

[0142] Use traditional non-learning methods such as bilateral interpolation and Laplace completion to fill in the gaps;

[0143] RGB-D image 3D reconstruction and projection to generate new depth maps.

[0144] The multimodal data synchronization verification module aligns multimodal data such as RGB images, depth images, and trajectory data at the timestamp level using alignment methods including: image frame content similarity comparison, semantic event alignment, and sensor synchronization signals; content consistency can be achieved through image content fingerprinting, intra-frame key feature point distribution, and image quality function comparison.

[0145] The enhanced data output module can output data in formats such as ROS Bag, PCL point cloud data, and database structure records; it also supports data tracking. The data tracking methods include, but are not limited to: using traditional version control mechanisms (such as Git structures) to achieve historical data recording and backtracking; and using a signed chained storage method to achieve data integrity verification and traceability.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the present invention.

Claims

1. A robotic multi-modal data augmentation system for industrial close-range grasping scenarios, characterized by: The system comprises a data preprocessing module, a background reconstruction module, an adaptive illumination enhancement module, a depth information reconstruction module, a multi-modal data synchronization verification module, and an enhanced data output module. The data preprocessing module acquires an HDF5 data format file composed of robot execution trajectory and visual information, and performs preprocessing. The background reconstruction module performs background reconstruction on the RGB image to enhance image diversity. The adaptive illumination enhancement module takes the background reconstructed RGB image output by the background reconstruction module as input, generates multiple versions of illumination disturbance images within the parameter range of brightness, contrast, and gamma value, and performs fusion processing on the multiple versions of illumination disturbance images based on a preset fusion strategy to obtain an enhanced RGB image with robustness under various illumination conditions. The depth information reconstruction module reconstructs and completes missing or invalid areas in the original depth image. The multi-modal data synchronization verification module is used to synchronize and verify the enhanced RGB image, the reconstructed and completed depth image, and the trajectory data, and generate corresponding verification codes to ensure the consistency of the RGB image, the depth image, and the trajectory data during the enhancement process. The enhanced data output module repackages the enhanced RGB image, the depth image, the trajectory data, and the corresponding verification codes and outputs them to form a data set that can be directly used for training or deployment.

2. The robot multi-modal data augmentation system for industrial close-range grasping scenarios according to claim 1, wherein: The data preprocessing module completes data preprocessing through the following steps: Perform integrity check and outlier detection on the HDF5 data format file. Extract the RGB image and depth image data in the HDF5 data format file, and associate the trajectory data based on timestamp alignment technology. Generate a hash verification code for each frame of data of the RGB image and the depth image for consistency verification.

3. The robotic multi-modal data augmentation system for industrial close-range grasping scenarios according to claim 2, wherein: The background reconstruction module comprises a SAM segmentation engine, a Gaussian background generator, and a scene reorganizer. The SAM segmentation engine automatically extracts the foreground region including the robot arm and the target object in the RGB image using the SAM algorithm. The Gaussian background generator performs Gaussian blur on the background region after RGB image segmentation to construct a uniform and controllable blurred background. The scene reorganizer seamlessly splices the blurred background and the foreground region to form a natural and realistic enhanced RGB image.

4. The robotic multi-modal data augmentation system for industrial close-range grasping scenarios according to claim 3, wherein: The SAM segmentation engine segments the input RGB image based on the SAM2 algorithm The foreground region is extracted, and the processing flow is as follows: First, the RGB image is subjected to image encoding to obtain features: ； wherein, denotes an image feature extraction network, are parameters of the image feature extraction network; is an encoded image feature representation; Then, position embedding items are added to the encoded image features: ； wherein, is an image feature after adding a position embedding term, is a position embedding matrix used to add spatial position information to the encoded feature map; Then, the image features with position embedding items are fused with prompt information through an attention mechanism, and a mask probability map is obtained through a mask decoding function: ； ； wherein, is a hint encoding function, is a hint encoding function parameter; denotes the input hint information; is a mask decoding function, is a mask decoding parameter; is the output mask probability map, denoting the probability that a pixel belongs to the foreground, 、 are the height and width of the mask probability map respectively. Finally, the foreground mask is obtained through thresholding: ； wherein, is a threshold parameter; when , the pixel point belongs to the foreground region, otherwise, the background region.

5. The robotic multi-modal data augmentation system for industrial close-range grasping scenarios according to claim 4, wherein: The adaptive illumination enhancement module introduces a random seed control mechanism to perform n times of disturbance within a 10% range on the brightness, contrast, and gamma value of the RGB image to generate different versions of RGB images.

6. The robotic multi-modal data augmentation system for industrial close-range grasping scenarios of claim 5, wherein: The specific method of the adaptive illumination enhancement module for disturbing the brightness, contrast, and gamma value of the RGB image is: A random seed value s is set, a perturbation coefficient in the range of [-0.1, +0.1] is generated based on a pseudo-random number generator, and the perturbation coefficient is used to perturb the brightness, contrast and gamma value of the RGB image; the calculation method is as follows: ； ； ； ； wherein, is an image perturbed for input RGB image for brightness and contrast, is an image perturbed for brightness and contrast perturbed image for gamma value, is a contrast adjustment coefficient, is a brightness offset, is a gamma correction coefficient; , , respectively denote contrast, brightness and gamma perturbation factors, subject to a uniform distribution within the interval wherein is a random seed used to control the reproducibility of the perturbation;​ The enhanced RGB image is formed by fusing the RGB images processed by different perturbation degrees through the hybrid policy executor, and the fusion formula is as follows: ； wherein, represents the pixel value of the RGB image obtained after the i-th perturbation at pixel (x, y); represents the weight of the RGB image obtained after the i-th perturbation, which is adaptively assigned based on the quality indicator; is the enhanced RGB image after fusion of the RGB images obtained after n perturbations.

7. The robotic multi-modal data augmentation system for industrial close-range grasping scenarios of claim 6, wherein: The depth information reconstruction module adopts a depth completion pre-training network model DenseDepth based on an encoder-decoder structure to perform depth prediction on the RGB-D image fused by the RGB image and the depth image in the system training stage, so as to realize the completion and fusion of the missing depth area of the depth image and improve the integrity and depth consistency of the training data; First, the RGB image and the depth image are fused to obtain an RGB-D image , , where H and W are the height and width of the RGB-D image, a pre-trained network model DenseDepth is used to perform depth prediction on the input RGB-D image to obtain a predicted depth map ; Then, the predicted depth map is fused with the original depth map in the missing region to obtain a completed depth map ; In the training process, the DenseDepth network is updated end to end by minimizing a plurality of joint loss functions: ； wherein, is a scale-invariant log error term, is a gradient smoothing term, is a structural similarity term, is a reverse Huber error term; 、 、 、 are weights for the respective loss terms.

8. The robotic multi-modal data augmentation system for industrial close-range grasping scenarios according to claim 7, wherein: The multi-modal data synchronous verification module includes a time sequence alignment checker and an integrity verifier; The time sequence alignment checker ensures that the multi-modal data such as the RGB image, the depth image and the trajectory data are aligned at the timestamp level; The integrity verifier compares the original hash check code of the RGB image and the depth image with the multi-modal data such as the enhanced RGB image and the reconstructed and completed depth image, and verifies the content consistency.

Images