A cross-modal correction fusion robot grasping detection method and system

Abstract

The application discloses a cross-modal correction fusion robot grabbing detection method and system, constructs a cross-modal correction fusion grabbing detection model based on edge guidance, collects a grabbing data set for simulating a production line environment, establishes an Xauat-Grasping single-target grabbing data set, carries out data enhancement on the public data set and the Xauat-Grasping single-target grabbing data set, and divides a training set and a test set; the cross-modal correction fusion grabbing detection model is trained by using the training set, the test set is used for testing in a real scene, and the cross-modal correction fusion robot grabbing detection is realized by using the tested cross-modal correction fusion grabbing detection model. While guaranteeing the real-time of grabbing, the grabbing precision is improved, and the very strong robustness is possessed under the complex environment.

Description

Technical Field

[0001] This invention belongs to the fields of image processing, deep learning, and robot grasping and detection technology, and specifically relates to a cross-modal correction and fusion robot grasping and detection method and system. Background Technology

[0002] In recent years, intelligent robot technology has developed rapidly in fields such as industry, agriculture, medical assistance, social services, and space exploration. Accurate object identification and grasping are fundamental to subsequent operations during robot task execution. Currently, robot grasping and detection technologies can be divided into analysis-based and data-driven methods. Analysis-based grasping and detection mainly relies on object geometry, physical models, kinematics, dynamics, and mechanical analysis to calculate the parameters of the target to be grasped. These methods are only applicable to specific tasks in specific environments and lack diverse working capabilities. Data-driven grasping and detection uses diverse data to improve the robot's grasping performance in various environments, tasks, and scenarios. Compared to analysis-based grasping and detection, data-driven methods do not require analysis of the target's geometric relationships or physical models, thus reducing computational complexity. Therefore, data-driven grasping and detection has received widespread attention in grasping problems involving unknown objects.

[0003] With the widespread application of RGB-D sensors, more and more researchers are turning their attention to the research of dual-modal fusion of RGB and depth images for grasping and detection. Multimodal fusion techniques can be divided into three types: early fusion (feature-level fusion), late fusion (decision-level fusion), and hybrid fusion. Although early fusion can better understand the relationships between features, it is difficult to achieve complete fusion because the fused features are far from the decoder. Late fusion can better handle the overfitting problem, but it is difficult to extract the intrinsic relationship between the two modalities. Hybrid fusion, although increasing the computation time, makes better use of multimodal information and effectively improves network performance. The fusion of RGB-D information can complement each other's color and depth information to obtain more effective grasping semantic features. However, the uneven quality of depth sensors and the characteristics of the grasped object itself (transparency, reflection) bring a lot of strong noise interference and information loss problems. These problems pose challenges to the grasping of unknown objects by networks based on conventional fusion methods. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a cross-modal correction and fusion robot grasping and detection method and system to address the shortcomings of the prior art. This method and system are used to solve the technical problem of insufficient robot grasping accuracy caused by noise in grasping information in complex industrial environments. The invention achieves mutual correction and fusion of RGB information and depth information to improve the quality of object grasping features.

[0005] The present invention adopts the following technical solution:

[0006] A cross-modal correction fusion robot grasping and detection method includes the following steps:

[0007] S1. Construct a cross-modal correction and fusion grasping and detection model based on edge guidance;

[0008] S2. Collect and extract datasets to simulate the production line environment, and establish a Xauat-Grasping single-target crawling dataset.

[0009] S3. Perform data augmentation on the public dataset and the Xauat-Grasping single-target crawling dataset obtained in step S2, and divide it into training and test sets;

[0010] S4. Use the training set obtained in step S3 to train the cross-modal correction fusion grasping and detection model obtained in step S1. Use the test set obtained in step S3 to test it in a real scene. Use the tested cross-modal correction fusion grasping and detection model to realize cross-modal correction fusion robot grasping and detection.

[0011] Specifically, in step S1, the edge-guided cross-modal correction fusion grasping and detection model includes an encoding stage, a decoding stage, and a grasping prediction stage.

[0012] The encoding stage constructs a bimodal feature extraction module, which consists of four stages. Each stage is composed of two parallel layers of regular convolutions and stacked FasterNet modules. Each FasterNet module is an inverted residual module formed by one partial convolution and two regular convolutions. After sampling at each stage, the feature enters the cross-modal correction and fusion module. The corrected features enter the next stage, and the fused features are fed to the upsampling stage step by step.

[0013] In the decoding stage, a Context module is constructed, which aggregates features at different scales by introducing multiple branches, and an upsampling module is used to fuse the rich semantic information of high-level features with the rich spatial information contained in low-level features.

[0014] In the grasp prediction phase, four sub-networks are used to predict the grasp quality (Q), grasp angle (θ), and grasp width (W) respectively. The grasp angle (θ) is predicted using two sub-networks, and then a third sub-network is used to guide the learning of the grasp object boundary.

[0015] Furthermore, in the encoding stage, the cross-modal correction fusion module is used to correct noise in the RGB and depth features from sequence to sequence, and then compensates for the information loss problem through fusion, defined as:

[0016]

[0017]

[0018]

[0019] in, and These are the data streams of the RGB image and the depth image after being corrected by the CCFM module during feature extraction, respectively. rgb and x depth These are unprocessed RGB image and depth image data streams, respectively.

[0020] Furthermore, during the decoding phase, the Context module processes the process as follows:

[0021] F h =Concat(Conv6(F x ),Conv 12 (F x ),Conv 18 (F x ))

[0022] The upsampling module is defined as follows:

[0023]

[0024] F a =F m ·view(Fuse)·+Fuse

[0025] F m =Softmax(view(Conv) 1×1 (F up )))

[0026] Among them, F up F is the feature map obtained from one upsampling. m This is an attention map, where Fuse represents low-level semantic features. a For the corrected feature map, F h For the aggregated high-level semantic features, Conv6, Conv 12 Conv 18 These are dilated convolutions with kernel size of 3 and dilation rates of 6, 12, and 18, respectively.

[0027] Furthermore, the process of using a sub-network to guide the learning of the object's boundary is as follows:

[0028]

[0029] Where Emap represents the learned edge-grabbing features, and Conv represents a task-specific convolution with a kernel of 1.

[0030] Specifically, in step S2, the grasping dataset includes the currently publicly available Cornell grasping detection dataset, which includes 85 images with a resolution of 640×480 and a total of 240 objects; the production line environment was simulated using the DOBOT CR3 robot and Realsense D435i depth camera to establish the Xauat-Grasping single-object grasping dataset, which contains 281 different objects and a total of 727 samples, each sample being 640×480 in size.

[0031] Specifically, in step S3, the rotation annotation tool roLabellmg is used to complete the annotation of the grabbable region and data augmentation on an image-by-image basis. Before training, the images are adjusted to 224×224. On the Cornell and Xauat-Grasping datasets, 90% are used for training and 10% are used for testing.

[0032] Specifically, in step S4, the training parameters are set with a batch size of 16, an epoch of 60, an initial learning rate of 0.0001, and the Adam optimizer is used for optimization.

[0033] Furthermore, the loss function for the training process is:

[0034] Loss=λ0L reg +λ1L edge

[0035] Where λ0 and λ1 represent the capture loss weight and the edge guidance strategy loss weight, respectively, L edge Let L be the cross-entropy loss function for binary classification. reg The loss function is smooth L1.

[0036] Secondly, embodiments of the present invention provide a cross-modal correction fusion robot grasping and detection system, comprising:

[0037] The module builds a cross-modal correction and fusion grasping and detection model based on edge guidance;

[0038] The environment module collects and scrapes datasets to simulate the production line environment and establishes the Xauat-Grasping single-target scraping dataset.

[0039] The data module performs data augmentation on the Xauat-Grasping single-target crawling dataset obtained from the public dataset and the environment module, and splits it into training and test sets;

[0040] The detection module uses the training set obtained from the data module to train the cross-modal correction and fusion grasping detection model obtained from the construction module, and uses the test set obtained from the data module to test it in a real-world scenario. The tested cross-modal correction and fusion grasping detection model is then used to achieve cross-modal correction and fusion robot grasping detection.

[0041] Compared with the prior art, the present invention has at least the following beneficial effects:

[0042] A cross-modal correction and fusion robot grasping detection method is proposed. This method generates a grasping target edge map using the Canny edge detection algorithm and guides the network to focus more on object edges during training, capturing more accurate grasping information. This solves the problem of local optima caused by the network focusing only on the grasping region and ignoring the grasping target itself during training. The method employs a cross-modal correction and fusion module (CCFM) that utilizes a cross-attention mechanism to correct noise in RGB and depth images from sequence to sequence. It also effectively addresses the problem of incomplete semantic information extracted by the network due to unclear feature information. Furthermore, the method utilizes a positional attention mechanism to ensure that the fused features retain more grasping position information.

[0043] Furthermore, in the encoding stage, the dual-channel feature extraction network, compared to the single-channel feature extraction network, considers the complementarity of RGB and depth images, exhibiting stronger robustness. However, the dual-channel extraction network needs to process two different input data, resulting in a large number of parameters and challenging real-time performance. Our proposed dual-modal extraction module in the encoding stage addresses both accuracy and real-time performance. RGB images provide color information of the target, while depth images provide spatial depth information. However, conventional fusion methods fail to fully utilize the complementarity between the two modalities, limiting the expressive power of the fused result. Moreover, direct or simple fusion methods may be overly sensitive to factors such as illumination, noise, and occlusion, reducing the robustness of the fusion. To address this issue, this paper proposes a CCFM module that adaptively corrects between different data streams in an end-to-end manner, more effectively utilizing the captured information. Using multi-level feature fusion in the encoding stage enriches feature representation, compensates for lost information, and improves the network's ability to understand the semantic information of the captured target.

[0044] Furthermore, the constructed Xauat-Grasping dataset covers common objects in daily life as well as challenging objects from industrial production, medical and chemical industries, such as transparent and textureless objects, non-transparent and textureless objects, and easily deformable objects. Studying the grasping performance of these objects in real-world scenarios has greater value for the correction and fusion of depth images and RGB images, as well as feature extraction capabilities.

[0045] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0046] In summary, this invention improves grasping accuracy while ensuring real-time grasping performance, and exhibits strong robustness in complex environments.

[0047] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the overall structure of the edge-guided robot grasping and detection model in an embodiment of the present invention;

[0049] Figure 2 A diagram illustrating the grabbing configuration representation method provided by this invention. (x, y) represents the grabbing center coordinates, θ represents the grabbing angle, width represents the grabbing width, and Q represents the grabbing quality.

[0050] Figure 3 This is a schematic diagram of the method flow of the present invention;

[0051] Figure 4 The environment diagram for constructing the dataset for this invention is shown, where (a) shows the structure of the robotic arm, the sensor, and the installation method of the end effector, and (b) shows the dataset captured.

[0052] Figure 5 The images show the test results of the edge-guided grasping and detection model on real objects in this embodiment of the invention. (a) and (b) are regular objects, (c) and (d) are textureless objects, (e) and (f) are transparent objects, and (g) and (h) are easily deformable objects.

[0053] Figure 6 A schematic diagram of a computer device provided in an embodiment of the present invention;

[0054] Figure 7 This is a block diagram of a chip provided according to an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0057] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0058] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0059] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0060] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0061] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0062] This invention provides a cross-modal correction and fusion robot grasping detection method, constructing an edge-guided cross-modal correction and fusion grasping detection model; combining edge sub-networks with edge prior knowledge, and using cross-entropy loss to supervise the network in learning the structural features of the grasping target, effectively improving the network's performance in object grasping detection; collecting grasping datasets, and using the DOBOT CR3 robot and Realsense D435i depth camera to simulate a production line environment, establishing a single-target grasping dataset named Xauat-Grasping; performing data augmentation on the publicly available Cornell and Xauat-Grasping single-target grasping datasets, and proportionally dividing them into training and test sets; training the model using a server, and testing its performance on the test set and in real-world scenarios.

[0063] Please see Figure 3 The present invention discloses a cross-modal correction fusion robot grasping and detection method, comprising the following steps:

[0064] S1. Construct a cross-modal correction and fusion grasping and detection model based on edge guidance;

[0065] Please see Figure 1 The edge-guided cross-modal correction fusion grasping and detection model consists of three stages: encoding, decoding, and grasping prediction. Specifically:

[0066] The encoding stage considers the complementarity of RGB and depth images, as well as the real-time requirements of grasping detection, and constructs a bimodal feature extraction module, which is divided into four stages. Each stage consists of two parallel layers of regular convolutions and stacked FasterNet modules. Each FasterNet module is an inverted residual module formed by one partial convolution (PConv) and two regular convolutions (with a kernel size of 1). The processing procedure is as follows:

[0067]

[0068]

[0069] Where, x rgb and x depth These are RGB image and depth image data streams, respectively, C rgb and C depth These are the data streams before feature extraction at each stage, where FB stands for FasterNet Block and PC stands for PC. onv For partial convolution, L is the number of stacked FBs.

[0070] After each sampling stage, the system enters the cross-modal correction and fusion module (CCFM). The corrected features then enter the next stage, and the fused features are then fed to the upsampling stage step by step.

[0071] To compensate for noise and depth information loss caused by sensors, a cross-modal correction fusion module (CCFM) is proposed. This module corrects noise in the RGB and depth features from sequence to sequence, and then uses fusion to compensate for the information loss. It is defined as follows:

[0072]

[0073]

[0074]

[0075] in, and These are the data streams of the RGB image and the depth image after being corrected by the CCFM module during feature extraction, respectively. rgb and x depth These are unprocessed RGB image and depth image data streams, respectively.

[0076] In order to enhance the reception domain of the encoding stage during the decoding stage, inspired by the pyramid pooling structure in PSPNet, this invention constructs a Context module, which aggregates features at different scales by introducing multiple branches. Secondly, a novel upsampling module DEC is designed to fuse the rich semantic information of high-level features with the rich spatial information contained in low-level features.

[0077] The Context module processes the data as follows:

[0078] F h =Concat(Conv6(F x ),Conv 12 (F x ),Conv 18 (F x ))

[0079] Among them, F h For the aggregated high-level semantic features, Conv6, Conv 12 Conv 18 These are dilated convolutions with kernel size of 3 and dilation rates of 6, 12, and 18, respectively.

[0080] The DEC module is defined as follows:

[0081]

[0082] F a=F m ·view(Fuse)·+Fuse

[0083] F m =Softmax(view(Conv) 1×1 (F up )))

[0084] Among them, F up F is the feature map obtained from one upsampling. m This is an attention map, where Fuse represents low-level semantic features. a This is the corrected feature map.

[0085] In the grasping prediction stage, unlike other detection methods, this invention uses five sub-networks. Four sub-networks are used to predict the grasping quality (Q), grasping angle (θ), and grasping width (W), respectively. Due to the symmetry of the angle, two sub-networks are used to calculate the sine and cosine of the grasping angle (θ) to obtain the grasping angle.

[0086] An additional subnetwork is used to guide the learning of the boundaries of the grasped object, and the process is as follows:

[0087]

[0088] Where Emap represents the learned edge-grabbing features, and Conv represents a task-specific convolution with a kernel of 1.

[0089] By combining edge subnetworks with edge prior knowledge and using cross-entropy loss to supervise the network in learning the structural features of the grasped target, the network's performance in object grasping and detection is effectively improved.

[0090] Please see Figure 2 The crawling is represented as:

[0091] Gr = {x, y, Q, W, θ}

[0092] Where Gr represents the pose vector output by the grasping network, (x,y) represents the center coordinates of the grasping box, W represents the opening and closing distance of the gripper with a pixel range of [0,150], and Q represents the grasping quality score of each pixel, with a value between 0 and 1. The closer to 1, the higher the probability of successful grasping.

[0093] Considering the symmetry of the grasping angle, i.e., it yields the same result at θ as it does at (θ+π), sin(2θ) and cos(2θ) are used to encode the actual rotation angle, and the calculation formula is as follows:

[0094]

[0095] Here, θ represents the rotation angle.

[0096] S2. Collect the grasping dataset and use the DOBOT CR3 robot and Realsense D435i depth camera to simulate the production line environment and establish a single-target grasping dataset called Xauat-Grasping.

[0097] The crawl dataset includes the currently publicly available Cornell crawl detection dataset, which consists of 885 images with a resolution of 640×480 and a total of 240 objects.

[0098] 5110 positive grasping labels and 2909 negative labels were manually annotated on these 240 objects. To further study the performance of robot grasping detection, this invention used the DOBOT CR3 robot and a Realsense D435i depth camera to simulate a production line environment, establishing a single-object grasping dataset called Xauat-Grasping. This dataset covers regular objects and some challenging objects, such as transparent objects without texture, non-transparent objects without texture, and easily deformable objects. The dataset contains 281 different objects, totaling 727 samples, each with a size of 640×480.

[0099] S3. Perform data augmentation on the public dataset Cornell and the lab-built dataset Xauat-Grasping, and use online random rotation, pruning and other methods to expand the data, and divide the training set and test set in a 9:1 ratio;

[0100] Image data preprocessing includes using the rotation annotation tool roLabellmg to annotate the graspable regions image by image, and using the program built in this invention to convert them into the grasping format labels required for robot grasping. Secondly, the images are adjusted to 224×224 before training. 90% of the data is used for training on the Cornell and Xauat-Grasping datasets, and 10% is used for testing.

[0101] Both datasets were augmented using online random rotation and cropping methods. The grabbable regions were labeled image-by-image using the rotation annotation tool roLabellmg, and then converted into the required grabbable label format using the program developed in this invention.

[0102] S4. Train the constructed edge-guided cross-modal correction fusion grasping and detection model and test it on the test set and in real-world scenarios.

[0103] The server used for training was equipped with an Intel Xeon E5 2650 processor, 128GB of RAM, and one NVIDIA GeForce RTX 2080Ti graphics card. The deep learning framework used was PyTorch-1.6, the NVIDIA CUDA 10.2 GPU platform, and the cuDNN 7.0 deep learning GPU acceleration library. During network training, the proposed network was configured with a batch size of 16, 60 epochs, and an initial learning rate of 0.0001, and optimized using the Adam optimizer.

[0104] An edge detection algorithm is employed to acquire prior knowledge of the edges of the target object being grasped. This aims to inject boundary-related edge cues into feature representation learning, thereby enhancing the ability to represent features with object structure. Therefore, an edge sub-network is combined with edge prior knowledge, and cross-entropy loss is used to supervise the network's learning of the structural features of the grasped target.

[0105] Define the binary classification cross-entropy loss function as:

[0106]

[0107] Where y is the ground truth edge label obtained from the original image after image processing. The probability is estimated for the margin.

[0108] Clearly, using edge prior knowledge as a guide enables the network to quickly focus on capturing complete information about the target. Simultaneously, by gradually reducing the proportion of edge prior knowledge, the network moves from capturing the complete object to focusing on the graspable region. This design better simulates human attention to edge features, thereby effectively improving the network's performance in object detection and grasping.

[0109] The loss function during training is defined as follows:

[0110] Loss=λ0L reg +λ1L edge

[0111] Where λ0 and λ1 represent the grasping loss weight and the edge guidance strategy loss weight, respectively. Let the grasping loss weight λ0 be a constant of 10, and let the edge guidance strategy loss weight λ1 gradually decrease as the number of training epochs increases. Its function is defined as:

[0112]

[0113] Where ω is the initial weight constant 8, e is the number of training rounds, and l is the decay round constant 12.

[0114] The regression loss is calculated using the smooth L1 loss function:

[0115]

[0116] in, To capture the estimated value, G m This corresponds to the tag value.

[0117] In another embodiment of the present invention, a cross-modal correction fusion robot grasping and detection system is provided. This system can be used to implement the above-mentioned cross-modal correction fusion robot grasping and detection method. Specifically, the cross-modal correction fusion robot grasping and detection system includes a construction module, an environment module, a data module, and a detection module.

[0118] Among them, the construction module builds a cross-modal correction and fusion grasping and detection model based on edge guidance;

[0119] The environment module collects and scrapes datasets to simulate the production line environment and establishes the Xauat-Grasping single-target scraping dataset.

[0120] The data module performs data augmentation on the Xauat-Grasping single-target crawling dataset obtained from the public dataset and the environment module, and splits it into training and test sets;

[0121] The detection module uses the training set obtained from the data module to train the cross-modal correction and fusion grasping detection model obtained from the construction module, and uses the test set obtained from the data module to test it in a real-world scenario. The tested cross-modal correction and fusion grasping detection model is then used to achieve cross-modal correction and fusion robot grasping detection.

[0122] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a cross-modal correction fusion robot grasping and detection method, including:

[0123] Construct an edge-guided cross-modal correction and fusion grasping detection model; collect grasping datasets to simulate a production line environment and establish a Xauat-Grasping single-target grasping dataset; perform data augmentation on the public dataset and the Xauat-Grasping single-target grasping dataset, and divide them into training and testing sets; train the cross-modal correction and fusion grasping detection model using the training set, test it in a real-world scenario using the testing set, and use the tested cross-modal correction and fusion grasping detection model to achieve cross-modal correction and fusion robot grasping detection.

[0124] Please see Figure 6 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the fluid composition calculation method in the reservoir stimulation wellbore of this embodiment. To avoid repetition, details are omitted here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the cross-modal correction fusion robot grasping and detection system of this embodiment. To avoid repetition, details are omitted here.

[0125] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 6This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0126] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0127] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0128] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0129] Please see Figure 7 The terminal device is a chip. In this embodiment, the chip 600 includes a processor 622, which may be one or more, and a memory 632 for storing computer programs executable by the processor 622. The computer program stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 622 may be configured to execute the computer program to perform the aforementioned cross-modal correction fusion robot grasping and detection method.

[0130] Additionally, chip 600 may also include a power supply component 626 and a communication component 650. The power supply component 626 can be configured to perform power management of chip 600, and the communication component 650 can be configured to enable communication of chip 600, such as wired or wireless communication. Furthermore, chip 600 may also include an input / output (I / O) interface 658. Chip 600 can operate on an operating system stored in memory 632.

[0131] In another embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0132] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the cross-modal correction fusion robot grasping and detection method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps:

[0133] Construct an edge-guided cross-modal correction and fusion grasping detection model; collect grasping datasets to simulate a production line environment and establish a Xauat-Grasping single-target grasping dataset; perform data augmentation on the public dataset and the Xauat-Grasping single-target grasping dataset, and divide them into training and testing sets; train the cross-modal correction and fusion grasping detection model using the training set, test it in a real-world scenario using the testing set, and use the tested cross-modal correction and fusion grasping detection model to achieve cross-modal correction and fusion robot grasping detection.

[0134] Please see Figure 4The robotic arm used in the grasping system is a DOBOT CR3. The depth camera is a RealSense D435i, and the end effector is a parallel gripper. The specific installation method is shown in Figure (a), simulating a production line environment. A single-object grasping dataset, Xauat-Grasping, covering regular objects and challenging objects (transparent without texture, non-transparent without texture, and easily deformable), was established. Figure (b) shows the RGB images for each class. Studying the grasping performance of these objects in real-world scenarios is of greater value for the correction and fusion of depth and RGB images, as well as feature extraction capabilities.

[0135] Please see Figure 5 The invention demonstrates the grasping results in a real-world scenario, achieving accurate grasping of both conventional and challenging objects, thus proving its effectiveness and robustness.

[0136] Qualitative and quantitative analyses on the Cornell and Xauat-Grasping datasets demonstrate that the proposed network achieves a good balance between crawling speed and accuracy. It achieves crawling detection success rates of 98.8% and 75.3% on the Cornell and Xauat-Grasping datasets, respectively.

[0137] In summary, the cross-modal correction fusion robot grasping and detection method and system of the present invention have the following characteristics:

[0138] To address the issue that generative crawling detection networks often get stuck in local optima because they ignore the overall characteristics of the target, a new approach is proposed that uses edge detection algorithms to generate prior knowledge of the target's edges, guiding the network to analyze the crawlable region from the complete information of the target.

[0139] To address the issues of depth information loss and superimposed noise interference in depth images, a cross-modal correction and fusion module (CAFM) was introduced. By performing cross-modal correction and fusion on the features of RGB and depth images, the network performance was further improved.

[0140] A real-world 2D grasping challenge dataset, Xauat-Grasping, was constructed, which includes regular objects, textureless objects, and easily deformable objects. By restoring the information loss and noise in depth images, it provides rich data resources for subsequent cross-modal fusion grasping research.

[0141] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0142] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0144] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0147] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random-access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0148] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A cross-modal correction fusion robot grasping and detection method, characterized in that, Includes the following steps: S1. Construct an edge-guided cross-modal correction fusion grasping and detection model, which includes an encoding stage, a decoding stage, and a grasping prediction stage. The encoding stage constructs a bimodal feature extraction module, which consists of four stages. Each stage is composed of two parallel layers of regular convolutions and stacked FasterNet modules. Each FasterNet module is an inverted residual module formed by one partial convolution and two regular convolutions. After sampling at each stage, the feature enters the cross-modal correction and fusion module. The corrected features enter the next stage, and the fused features are fed to the upsampling stage step by step. In the decoding stage, a Context module is constructed, which aggregates features at different scales by introducing multiple branches, and an upsampling module is used to fuse the rich semantic information of high-level features with the rich spatial information contained in low-level features. In the crawling prediction phase, four sub-networks are used to predict the quality of the crawling. Q ), grab angle ( θ ), Crawling width ( W ), including the grab angle ( θ Two subnetworks are used, and then a third subnetwork is used to guide the learning of the boundaries of the grasped object; During the encoding phase, the cross-modal correction and fusion module utilizes a cross-attention mechanism to correct noise in the RGB and deep features from sequence to sequence. Then, through fusion, it compensates for information loss, defined as follows: in, and These are the data streams of the RGB image and the depth image after being corrected by the cross-modal correction and fusion module during feature extraction. and These are the unprocessed RGB image and depth image data streams, respectively, and the loss function for the training process. for: in, , These represent the capture loss weight and the edge guidance strategy loss weight, respectively. The cross-entropy loss function is used for binary classification. The loss function is smooth L1. S2. Collect and extract datasets to simulate the production line environment, and establish a Xauat-Grasping single-target crawling dataset. S3. Perform data augmentation on the public dataset and the Xauat-Grasping single-target crawling dataset obtained in step S2, and divide it into training and test sets; S4. Use the training set obtained in step S3 to train the cross-modal correction fusion grasping and detection model obtained in step S1. Use the test set obtained in step S3 to test it in a real scene. Use the tested cross-modal correction fusion grasping and detection model to realize cross-modal correction fusion robot grasping and detection.

2. The cross-modal correction fusion robot grasping and detection method according to claim 1, characterized in that, During the decoding phase, the Context module processes the code as follows: The upsampling module is defined as follows: in, This is the feature map obtained from one upsampling. For attention maps, Low-level semantic features The corrected feature map, These are the high-level semantic features after aggregation. , , These are dilated convolutions with kernel size of 3 and dilation rates of 6, 12, and 18, respectively.

3. The cross-modal correction fusion robot grasping and detection method according to claim 1, characterized in that, The process of learning the boundaries of the grasped object using a sub-network is as follows: in, To learn edge features, This is a task-specific convolution with a kernel of 1.

4. The cross-modal correction fusion robot grasping and detection method according to claim 1, characterized in that, In step S2, the grasping dataset includes the currently available Cornell grasping detection dataset, which includes 85 images with a resolution of 640×480 and a total of 240 objects; the production line environment was simulated using the DOBOT CR3 robot and Realsense D435i depth camera to establish the Xauat-Grasping single-object grasping dataset, which contains 281 different objects and a total of 727 samples, each sample being 640×480 in size.

5. The cross-modal correction fusion robot grasping and detection method according to claim 1, characterized in that, In step S3, the rotation annotation tool roLabellmg is used to complete the annotation of the grabbable region and data augmentation on an image-by-image basis. Before training, the images are adjusted to 224×224. On the Cornell and Xauat-Grasping datasets, 90% are used for training and 10% are used for testing.

6. The cross-modal correction fusion robot grasping and detection method according to claim 1, characterized in that, In step S4, the training parameters are set with a batch size of 16, an epoch of 60, and an initial learning rate of 0.0001. The Adam optimizer is used for optimization.

7. A cross-modal correction fusion robot grasping and detection system, characterized in that, The cross-modal correction fusion robot grasping and detection method according to claim 1 includes: The module builds a cross-modal correction and fusion grasping and detection model based on edge guidance; The environment module collects and scrapes datasets to simulate the production line environment and establishes the Xauat-Grasping single-target scraping dataset. The data module performs data augmentation on the Xauat-Grasping single-target crawling dataset obtained from the public dataset and the environment module, and splits it into training and test sets; The detection module uses the training set obtained from the data module to train the cross-modal correction and fusion grasping detection model obtained from the construction module, and uses the test set obtained from the data module to test it in a real-world scenario. The tested cross-modal correction and fusion grasping detection model is then used to achieve cross-modal correction and fusion robot grasping detection.

Images