Deep learning visual grabbing method for transparent medical waste

Abstract

The invention belongs to the technical field of machine vision, and more particularly relates to a deep learning visual grabbing method for transparent medical wastes, which generates a three-channel composite image capable of capturing the characteristics of the transparent wastes through multi-modal data acquisition of RGB, a depth camera and an infrared camera, generates different simulation data through data enhancement, and obtains the characteristics of the transparent wastes. A multi-branch convolutional neural network model is adopted, and each branch independently extracts feature information of RGB, depth and infrared images in a shallow network. The deep network is combined with an attention mechanism, the edge recognition effect of the transparent object is emphatically optimized, and the boundary positioning precision of the transparent object is enhanced. Particularly, a special transparent object edge detection module is added, pseudo boundary interference caused by light reflection and refraction is effectively reduced, the trained model automatically detects the boundary and the grabbing point of the transparent medical waste, the robot screens out the most stable grabbing point through a preset threshold value, and therefore high-precision and stable automatic grabbing is achieved.

Description

Technical Field

[0001] The present application belongs to the field of machine vision technology, and more specifically, to a deep learning visual capture method for transparent medical waste. Background Art

[0002] In medical environments, medical waste includes various types of syringes, infusion bottles, blood collection tubes and other disposable medical supplies, many of which are transparent or translucent. These transparent medical wastes face high risks when handling because they may contain infectious materials or sharp objects, and direct manual operation is neither safe nor efficient. Therefore, the realization of automated transparent medical waste grabbing technology can not only help to increase the speed of medical waste treatment, but also reduce the risk of infection for operators, and achieve a contactless and safe treatment method.

[0003] Visual inspection of transparent medical waste presents unique challenges due to the particularity of its material and optical properties, mainly reflected in:

[0004] Blurred edges: Transparent objects present low contrast in images. Especially under complex lighting conditions, the edges of objects are often blurred and difficult to detect clearly using traditional image processing methods.

[0005] Light reflection and refraction: Medical waste may experience multiple light refraction and reflection phenomena on transparent surfaces, which may obscure the actual boundaries of objects or generate pseudo boundaries, thereby increasing the complexity of edge recognition.

[0006] However, due to the blurred edges and problems with light emission and refraction, technical problems such as inaccurate detection of transparent medical waste and unstable grasping occur. Summary of the invention

[0007] The present invention provides a deep learning visual capture method for transparent medical waste, which is intended to solve the current technical problems of inaccurate detection and unstable capture of transparent medical waste.

[0008] The deep learning visual capture method for transparent medical waste includes the following steps:

[0009] Step 1: Collect multimodal data based on RGB, depth camera and infrared camera;

[0010] Step 2: align the images captured by the RGB, depth camera and external infrared camera to generate three-channel composite image data; and generate different simulation data through data augmentation;

[0011] Step 3: A multi-branch convolutional neural network model is used. Each branch extracts feature information of different modalities on the shallow network. The deep network further optimizes the recognition effect of transparent object edges through the attention mechanism, and introduces a transparent object edge detection module to assist in edge positioning of transparent objects.

[0012] Step 4: Use the trained multi-branch convolutional neural network model to detect the boundaries and appropriate grasping points of transparent medical waste, and the robot automatically selects the most stable grasping point based on the preset threshold.

[0013] The present invention solves the problems of inaccurate detection and unstable grasping of transparent medical waste through a multimodal visual grasping method based on deep learning. First, through multimodal data collection of RGB, depth camera and infrared camera, a three-channel composite image that can capture the characteristics of transparent waste is generated. Through data enhancement, different simulation data are generated to further enrich the training data. Secondly, a multi-branch convolutional neural network model is adopted, and each branch independently extracts the feature information of RGB, depth and infrared images in the shallow network. The deep network is combined with the attention mechanism to focus on optimizing the edge recognition effect of transparent objects and enhance the boundary positioning accuracy of transparent objects. In particular, a special transparent object edge detection module is added to effectively reduce the pseudo-boundary interference caused by light reflection and refraction. Finally, the trained model automatically detects the boundaries and grasping points of transparent medical waste, and the robot selects the most stable grasping points through a preset threshold, thereby realizing high-precision and stable automated grasping.

[0014] Preferably, the specific steps of registering the images captured by the RGB, depth camera and external infrared camera are as follows:

[0015] Feature extraction: SIFT feature detection algorithm is used to extract key feature points from RGB, depth and infrared images to obtain feature points and feature vectors in each image;

[0016] Feature point matching: Using Euclidean distance judgment, feature point matching is performed between RGB images and other modality images to find point pairs with similar features:

[0017]

[0018] Where: f R represents the feature vector extracted from the RGB image; f M represents a feature vector found in other modality images, where the other modality images are depth camera images or infrared camera images;

[0019] If the Euclidean distance between two point pairs is less than the preset threshold, the two point pairs are considered to be feature point pairs;

[0020] Registration-affine transformation: Based on the matched feature point pairs, an affine transformation is established to convert the feature points in the RGB image into the coordinate system of the depth or infrared image:

[0021]

[0022] Where: (x R ,y R ) represents the coordinates of the feature points in the RGB image; (x M ,y M ) represents the coordinates of the corresponding feature points in the depth or infrared image; a, b, c, d represent scaling and rotation parameters; (t x ,t y ) represents the translation parameter;

[0023] Solve the radial transformation parameters: Solve the affine transformation parameters by the least squares method to minimize the error. After N pairs of feature point registration, establish the following minimization problem:

[0024]

[0025] Taking the derivative of the above minimization problem and setting the derivative to zero, we get the following linear equations:

[0026]

[0027] By solving the linear equations, we can get the radiation transformation parameters a, b, c, d, t x ,t y , complete the registration;

[0028] Generate three-channel composite image data: Combine the registered RGB, depth, and infrared images into a three-channel composite image.

[0029] Preferably, the specific steps of feature extraction are as follows:

[0030] Image preprocessing: Perform multi-resolution scaling on RGB, depth, and infrared images to construct images at multiple resolution levels; convert the scaled RGB images into grayscale images while retaining the grayscale characteristics of the depth image and infrared image. The grayscale conversion formula is as follows:

[0031] Igray=0.2989×I R +0.5870×I G +0.1140×I B ;

[0032] Where: I gray Represents a grayscale image; I R ,I G ,I BRepresents the red, green, and blue channels of an RGB image;

[0033] Construct a multi-scale Gaussian pyramid: For each level of resolution image I r Apply Gaussian blur to form a multi-scale space; generate a series of Gaussian blurred images through different Gaussian blur parameters:

[0034]

[0035] Where: G(x, y, σ) represents the Gaussian blurred image; x, y represent the pixel position; σ represents the blur scale parameter, which determines the blur intensity; e represents a mathematical constant or a natural constant;

[0036] A set of images with different blur scales are generated on each level of resolution to form a pyramid structure. Assuming that the image in the Gaussian scale space is L(x, y, σ), the scale pyramid of each level is expressed as:

[0037] L(x,y,σ)=G(x,y,σ)*I r ;

[0038] Where: L(x, y, σ) represents images at different scales; * represents convolution operation;

[0039] Construct a differential Gaussian pyramid: In each level of the Gaussian pyramid, adjacent fuzzy scale images are differentiated to construct a differential Gaussian pyramid to detect significant feature areas in the image:

[0040] D(x,y,σ)=L(x,y,kσ)-L(x,y,σ);

[0041] Where: D(x, y, σ) represents the differential Gaussian image; k represents the scale factor, which is used to generate images of different scales;

[0042] In the differential Gaussian image of each level of resolution, a 3×3×3 neighborhood extreme value detection is performed on each pixel, and the maximum or minimum value points are detected in the space and scale domains, and the maximum or minimum value points are marked as candidate key points;

[0043] Multi-resolution key point positioning and filtering: Accurately locate multi-resolution candidate key points, remove low-contrast feature points and unstable edge points, and optimize the key point position through Taylor expansion:

[0044]

[0045] Where: Represents the gradient at the key point; represents the Hessian matrix;

[0046] Determine the edge response value based on the Hessian matrix, filter out edge points, and ensure that the key point set is concentrated in the stable area;

[0047] Direction assignment: Gradient assignment and direction are calculated within the neighborhood of each multi-resolution keypoint:

[0048]

[0049]

[0050] Where: m(x,y) represents the gradient assignment; θ(x,y) represents the direction angle;

[0051] Construct a direction histogram for the neighborhood of each key point, select the main direction and assign it to the key point to achieve rotation invariance;

[0052] Feature vector generation and multi-resolution fusion: For each key point of each resolution, a 128-dimensional feature vector is generated, the neighborhood is divided into blocks, and the weighted gradients in 8 directions are calculated in each block to obtain the feature descriptor;

[0053] The feature points and descriptors of all resolution levels are fused by weighted averaging, where the weights are assigned according to the importance of each resolution level;

[0054] The feature vector obtained by normalizing the feature fusion is used to obtain the final extracted feature points and the feature vectors corresponding to the feature points.

[0055] Preferably, the generating of different simulation data includes performing data enhancement on the generated three-channel composite image to generate simulation data, wherein the data enhancement includes: performing random angle rotation on the composite image, scaling the composite image proportionally, performing random translation on the composite image, adding Gaussian noise to the composite image, or mirror flipping the composite image, or a combination of these.

[0056] Preferably, the multi-branch convolutional neural network model includes an input layer, a shallow branch feature extraction network, a deep attention mechanism network and a transparent object edge detection module;

[0057] Input layer: used to input three-channel composite image data, including RGB, depth and infrared channels. The input image size is H×W×3, where H represents height, W represents width, and 3 represents 3 channels.

[0058] The shallow branch feature extraction network includes RGB image branch, depth image branch and infrared image branch. The three branches are independent of each other and are used to extract feature information of RGB, depth and infrared images respectively. Each branch shares the convolutional network structure, but the parameters are not shared, as follows:

[0059] RGB branch:

[0060] Input: RGB channel data;

[0061] Convolutional layer 1: The convolution kernel size is 3×3, the stride is 1, the number of output channels is 64, and the feature map of H×W×64 is obtained;

[0062] Pooling layer 1: It is the maximum pooling layer with a window size of 2×2, a stride of 2, and an output size of The feature map of

[0063] Convolutional layer 2: The convolution kernel size is 3×3, the stride is 1, and the number of output channels is 128. The feature map of

[0064] Pooling layer 2: It is the maximum pooling layer with a window size of 2×2, a stride of 2, and an output size of The feature map of

[0065] Depth image branch:

[0066] Input: depth channel data;

[0067] The structure is the same as the RGB branch, and the convolution and pooling operations are performed continuously to obtain the output size The feature map of

[0068] Infrared imaging branch:

[0069] Input: infrared channel data;

[0070] The structure is the same as the RGB branch, and the convolution and pooling operations are performed continuously to obtain the output size The feature map of

[0071] Shallow branch output: The outputs of the three branches are concatenated in the channel dimension to obtain Composite feature map of

[0072] Deep Attention Mechanism Network:

[0073] Convolutional layer: The input is a composite feature map, the convolution kernel size is 3×3, the stride is 1, the number of output channels is 256, and the output is The feature map of

[0074] Attention mechanism module:

[0075] Input: feature map output by the convolutional layer;

[0076] Channel attention: weight each channel and generate a weighted feature map;

[0077] Transparent object edge detection module: including edge detection convolution and grasping point detection output layer;

[0078] The edge detection convolution includes:

[0079] Convolutional layer: The input is the output feature of the deep attention mechanism, the convolution kernel size is 3×3, the stride is 1, the number of output channels is 128, the activation function uses the ReLU activation function, and the output size is

[0080] Grasp point detection output layer:

[0081] Input: feature map after edge detection;

[0082] Convolution kernel size: 1×1, stride 1, number of output channels 2; output size is That is, the probability map of the grasping point position, which is the probability distribution of x and y coordinates respectively;

[0083] Softmax activation: The Softmax activation function is used on the grasping point position probability map to obtain the probability distribution, which is used to finally select the most appropriate grasping point.

[0084] Preferably, the loss function of the multi-branch convolutional neural network model includes edge detection loss, coordinate positioning loss and grasping point classification loss, as follows:

[0085]

[0086] Where: represents the edge detection loss; Indicates coordinate positioning loss; represents the classification loss of the grasping point; λ, γ, α, β all represent weight hyperparameters;

[0087]

[0088] Where: N represents the number of edge pixels; Represents the true edge label of the i-th pixel. If it belongs to the edge side, Otherwise, 0; Represents the edge probability value of the i-th pixel predicted by the model, with a value between [0,1];

[0089]

[0090] Where: M represents the number of grasping points; represents the coordinates of the jth real grasping point; represents the coordinates of the j-th grasp point predicted by the model;

[0091]

[0092] Where: represents the true label of the jth grasping point. If it is a graspable point, Otherwise, 0; It represents the graspability probability of the j-th grasp point predicted by the model, and its value is in the range of [0,1].

[0093] The beneficial effects of the present invention include:

[0094] The present invention solves the problems of inaccurate detection and unstable grasping of transparent medical waste through a multimodal visual grasping method based on deep learning. First, through multimodal data collection of RGB, depth camera and infrared camera, a three-channel composite image that can capture the characteristics of transparent waste is generated. Through data enhancement, different simulation data are generated to further enrich the training data. Secondly, a multi-branch convolutional neural network model is adopted, and each branch independently extracts the feature information of RGB, depth and infrared images in the shallow network. The deep network is combined with the attention mechanism to focus on optimizing the edge recognition effect of transparent objects and enhance the boundary positioning accuracy of transparent objects. In particular, a special transparent object edge detection module is added to effectively reduce the pseudo-boundary interference caused by light reflection and refraction. Finally, the trained model automatically detects the boundaries and grasping points of transparent medical waste, and the robot selects the most stable grasping points through a preset threshold, thereby realizing high-precision and stable automated grasping. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0096] Figure 1 An overall step block diagram provided for an embodiment of the present invention.

[0097] Figure 2 A schematic diagram of the structure of a multi-branch convolutional neural network model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0098] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0099] See also Figure 1 As shown, the best embodiment of the present invention is further described;

[0100] The deep learning visual capture method for transparent medical waste includes the following steps:

[0101] Step 1: Collect multimodal data based on RGB, depth camera and infrared camera;

[0102] Step 2: align the images captured by the RGB, depth camera and external infrared camera to generate three-channel composite image data; and generate different simulation data through data augmentation;

[0103] Step 3: A multi-branch convolutional neural network model is used. Each branch extracts feature information of different modalities on the shallow network. The deep network further optimizes the recognition effect of transparent object edges through the attention mechanism, and introduces a transparent object edge detection module to assist in edge positioning of transparent objects.

[0104] Step 4: Use the trained multi-branch convolutional neural network model to detect the boundaries and appropriate grasping points of transparent medical waste, and the robot automatically selects the most stable grasping point based on the preset threshold.

[0105] The present invention solves the problems of inaccurate detection and unstable grasping of transparent medical waste through a multimodal visual grasping method based on deep learning. First, through multimodal data collection of RGB, depth camera and infrared camera, a three-channel composite image that can capture the characteristics of transparent waste is generated. Through data enhancement, different simulation data are generated to further enrich the training data. Secondly, a multi-branch convolutional neural network model is adopted, and each branch independently extracts the feature information of RGB, depth and infrared images in the shallow network. The deep network is combined with the attention mechanism to focus on optimizing the edge recognition effect of transparent objects and enhance the boundary positioning accuracy of transparent objects. In particular, a special transparent object edge detection module is added to effectively reduce the pseudo-boundary interference caused by light reflection and refraction. Finally, the trained model automatically detects the boundaries and grasping points of transparent medical waste, and the robot selects the most stable grasping points through a preset threshold, thereby realizing high-precision and stable automated grasping.

[0106] As a possible implementation of this embodiment, the specific steps of registering the images captured by the RGB, depth camera and external infrared camera are as follows:

[0107] Feature extraction: SIFT feature detection algorithm is used to extract key feature points from RGB, depth and infrared images to obtain feature points and feature vectors in each image;

[0108] Feature point matching: Using Euclidean distance judgment, feature point matching is performed between RGB images and other modality images to find point pairs with similar features:

[0109]

[0110] Where: f R represents the feature vector extracted from the RGB image; f M represents a feature vector found in other modality images, where the other modality images are depth camera images or infrared camera images;

[0111] If the Euclidean distance between two point pairs is less than the preset threshold, the two point pairs are considered to be feature point pairs;

[0112] Registration-affine transformation: Based on the matched feature point pairs, an affine transformation is established to convert the feature points in the RGB image into the coordinate system of the depth or infrared image:

[0113]

[0114] Where: (x R ,y R ) represents the coordinates of the feature points in the RGB image; (x M ,y M ) represents the coordinates of the corresponding feature points in the depth or infrared image; a, b, c, d represent scaling and rotation parameters; (t x ,t y ) represents the translation parameter;

[0115] Solve the radial transformation parameters: Solve the affine transformation parameters by the least squares method to minimize the error. After N pairs of feature point registration, establish the following minimization problem:

[0116]

[0117] Taking the derivative of the above minimization problem and setting the derivative to zero, we get the following linear equations:

[0118]

[0119] By solving the linear equations, we can get the radiation transformation parameters a, b, c, d, t x ,t y , complete the registration;

[0120] Generate three-channel composite image data: Combine the registered RGB, depth, and infrared images into a three-channel composite image.

[0121] In this embodiment, the registration of three modal images, RGB, depth and infrared, can make up for the shortcomings of single modal images in feature capture. RGB images provide color information, depth images provide the three-dimensional form of objects, and infrared images are outstanding in low-light environments. This combination enhances the feature capture capability of transparent objects and helps to more comprehensively identify the edges and shapes of objects.

[0122] Affine transformation is performed based on feature point matching, and the transformation parameters are optimized through the least squares method, so that different modal images can be accurately aligned in the same coordinate system. The precise registration of multimodal images can generate clearer and more accurate three-channel composite images, improving the recognition effect of the edges of transparent medical waste.

[0123] As a possible implementation of this embodiment, the specific steps of feature extraction are as follows:

[0124] Image preprocessing: Perform multi-resolution scaling on RGB, depth, and infrared images to construct images at multiple resolution levels; convert the scaled RGB images into grayscale images while retaining the grayscale characteristics of the depth image and infrared image. The grayscale conversion formula is as follows:

[0125] I gray =0.2989×I R +0.5870×I G +0.1140×I B ;

[0126] Where: I gray Represents a grayscale image; I R ,I G ,I B Represents the red, green, and blue channels of an RGB image;

[0127] Construct a multi-scale Gaussian pyramid: For each level of resolution image I r Apply Gaussian blur to form a multi-scale space; generate a series of Gaussian blurred images through different Gaussian blur parameters:

[0128]

[0129] Where: G(x, y, σ) represents the Gaussian blurred image; x, y represent the pixel position; σ represents the blur scale parameter, which determines the blur intensity; e represents a mathematical constant or a natural constant;

[0130] A set of images with different blur scales are generated on each level of resolution to form a pyramid structure. Assuming that the image in the Gaussian scale space is L(x, y, σ), the scale pyramid of each level is expressed as:

[0131] L(x,y,σ)=G(x,y,σ)*I r ;

[0132] Where: L(x, y, σ) represents images at different scales; * represents convolution operation;

[0133] Construct a differential Gaussian pyramid: In each level of the Gaussian pyramid, adjacent fuzzy scale images are differentiated to construct a differential Gaussian pyramid to detect significant feature areas in the image:

[0134] D(x,y,σ)=L(x,y,kσ)-L(x,y,σ);

[0135] Where: D(x, y, σ) represents the differential Gaussian image; k represents the scale factor, which is used to generate images of different scales;

[0136] In the differential Gaussian image of each level of resolution, a 3×3×3 neighborhood extreme value detection is performed on each pixel, and the maximum or minimum value points are detected in the space and scale domains, and the maximum or minimum value points are marked as candidate key points;

[0137] Multi-resolution key point positioning and filtering: Accurately locate multi-resolution candidate key points, remove low-contrast feature points and unstable edge points, and optimize the key point position through Taylor expansion:

[0138]

[0139] Where: Represents the gradient at the key point; represents the Hessian matrix;

[0140] Determine the edge response value based on the Hessian matrix, filter out edge points, and ensure that the key point set is concentrated in the stable area;

[0141] Direction assignment: Gradient assignment and direction are calculated within the neighborhood of each multi-resolution keypoint:

[0142]

[0143] Where: m(x, y) represents the gradient assignment; θ(x, y) represents the direction angle;

[0144] Construct a direction histogram for the neighborhood of each key point, select the main direction and assign it to the key point to achieve rotation invariance;

[0145] Feature vector generation and multi-resolution fusion: For each key point of each resolution, a 128-dimensional feature vector is generated, the neighborhood is divided into blocks, and the weighted gradients in 8 directions are calculated in each block to obtain the feature descriptor;

[0146] The feature points and descriptors of all resolution levels are fused by weighted averaging, where the weights are assigned according to the importance of each resolution level;

[0147] The feature vector obtained by normalizing the feature fusion is used to obtain the final extracted feature points and the feature vectors corresponding to the feature points.

[0148] In this embodiment, the multi-scale Gaussian pyramid is used to extract features of different resolutions, which helps to capture the salient areas of the object at different scales. This multi-scale feature extraction method can perform multi-level modeling of the edge and morphological information of transparent objects, thereby improving the robustness of transparent object features.

[0149] Extremum detection in the differential Gaussian pyramid helps to accurately locate the significant feature points of transparent objects. By detecting the extremum in the space and scale domains, the stability and accuracy of the feature points at different resolutions are ensured, and the feature recognition error caused by the blurred edge of transparent objects is reduced.

[0150] In the multi-resolution feature extraction process, the Hessian matrix is ​​used to remove unstable edge points, ensuring that the extracted feature points are concentrated in the stable area of ​​the transparent object. This feature point optimization process effectively improves the accuracy of accurate positioning of the edge of the transparent object and the accuracy of feature extraction.

[0151] The weighted average fusion of multi-resolution feature points helps to obtain a more comprehensive feature description at different scales. By weighting the importance of different resolution levels, the fused feature vector is more comprehensive and stable, and can effectively deal with the edge blur problem of transparent objects in different environments.

[0152] The rotation invariance assignment of feature points makes the extracted feature vector highly robust and can adapt to the placement and changes of transparent objects at different angles. In addition, the generation of 128-dimensional feature vectors and multi-resolution fusion ensure the high accuracy of feature point description, making the final transparent object recognition and capture more stable and accurate.

[0153] As a possible implementation of this embodiment, the specific steps of fusing the feature points and descriptors at all resolution levels by using the weighted average method are as follows:

[0154]

[0155] Where: f(i) represents the importance function of resolution i to the feature descriptor; w i represents the weight of the i-th resolution level; d i represents the feature descriptor of the i-th resolution level; L represents the total number of resolution levels; d final Represents the final fused feature descriptor, which is a 128-dimensional vector;

[0156]

[0157] Where: SNR irepresents the signal-to-noise ratio of the i-th resolution, evaluated by contrast and noise level; σ i Represents the standard deviation of the i-th resolution level.

[0158] In this embodiment, by introducing f(i) to adjust the importance of different resolution levels, it is possible to better cope with changes in image noise and features at different resolutions. In particular, at low resolution, the contribution may be small due to fewer details. In this case, the weight of the resolution can be reduced, while when the details at high resolution are richer, it can be given a greater weight.

[0159] In some scenes, some key points at different resolutions are more important than other key points. By reflecting the contribution of these key points through weighting functions, the fusion of feature descriptors can be further optimized. And it allows the weight of each resolution to be dynamically adjusted according to the quality of the image (such as signal-to-noise ratio), so that high-quality images can make a greater contribution to the final fusion result, while low-quality images have less impact on the result.

[0160] As a possible implementation method of this embodiment, generating different simulation data includes performing data enhancement on the generated three-channel composite image to generate simulation data, wherein the data enhancement includes: rotating the composite image at a random angle, scaling the composite image proportionally, randomly translating the composite image, adding Gaussian noise to the composite image, or mirroring the composite image, or a combination of these.

[0161] See also Figure 2 As shown, as a possible implementation of this embodiment, the multi-branch convolutional neural network model includes an input layer, a shallow branch feature extraction network, a deep attention mechanism network and a transparent object edge detection module;

[0162] Input layer: used to input three-channel composite image data, including RGB, depth and infrared channels. The input image size is H×W×3, where H represents height, W represents width, and 3 represents 3 channels.

[0163] The shallow branch feature extraction network includes RGB image branch, depth image branch and infrared image branch. The three branches are independent of each other and are used to extract feature information of RGB, depth and infrared images respectively. Each branch shares the convolutional network structure, but the parameters are not shared, as follows:

[0164] RGB branch:

[0165] Input: RGB channel data;

[0166] Convolutional layer 1: The convolution kernel size is 3×3, the stride is 1, the number of output channels is 64, and the feature map of H×W×64 is obtained;

[0167] Pooling layer 1: It is the maximum pooling layer with a window size of 2×2, a stride of 2, and an output size of The feature map of

[0168] Convolutional layer 2: The convolution kernel size is 3×3, the stride is 1, and the number of output channels is 128. The feature map of

[0169] Pooling layer 2: It is the maximum pooling layer with a window size of 2×2, a stride of 2, and an output size of The feature map of

[0170] Depth image branch:

[0171] Input: depth channel data;

[0172] The structure is the same as the RGB branch, and the convolution and pooling operations are performed continuously to obtain the output size The feature map of

[0173] Infrared imaging branch:

[0174] Input: infrared channel data;

[0175] The structure is the same as the RGB branch, and the convolution and pooling operations are performed continuously to obtain the output size The feature map of

[0176] Shallow branch output: The outputs of the three branches are concatenated in the channel dimension to obtain Composite feature map of

[0177] Deep Attention Mechanism Network:

[0178] Convolutional layer: The input is a composite feature map, the convolution kernel size is 3×3, the stride is 1, the number of output channels is 256, and the output is The feature map of

[0179] Attention mechanism module:

[0180] Input: feature map output by the convolutional layer;

[0181] Channel attention: weight each channel and generate a weighted feature map;

[0182] Transparent object edge detection module: including edge detection convolution and grasping point detection output layer;

[0183] The edge detection convolution includes:

[0184] Convolutional layer: The input is the output feature of the deep attention mechanism, the convolution kernel size is 3×3, the stride is 1, the number of output channels is 128, the activation function uses the ReLU activation function, and the output size is

[0185] Grasp point detection output layer:

[0186] Input: feature map after edge detection;

[0187] Convolution kernel size: 1×1, stride 1, number of output channels 2; output size is That is, the probability map of the grasping point position, which is the probability distribution of x and y coordinates respectively;

[0188] Softmax activation: The Softmax activation function is used on the grasping point position probability map to obtain the probability distribution, which is used to finally select the most appropriate grasping point.

[0189] In this embodiment, by combining RGB, depth and infrared data, multi-dimensional feature information is extracted, which can more effectively distinguish the edges and background of transparent objects. This multi-modal input design makes up for the recognition defects of transparent object features under a single mode, and further optimizes the robustness of the model under different ambient lighting and transparency conditions.

[0190] Through independent RGB, depth, and infrared branches, we ensure that each modality feature is optimally extracted, reducing interference in the information fusion process. At the same time, each branch shares the convolution structure but not the parameters in the shallow network, which improves the pertinence and specificity of each modality feature extraction, helps generate high-quality composite feature maps, and optimizes the overall detection effect of the model.

[0191] The channel attention mechanism is introduced into the deep network, which weights the importance of different channels in the feature map to ensure that the most relevant edge features receive higher attention. This mechanism effectively reduces the interference of background noise and enhances the recognition ability of the edges of transparent objects, thereby improving the accuracy of grasping point positioning.

[0192] The edge detection module is specifically used to extract edge features of transparent objects. Combined with the output of the deep attention mechanism, the model can locate the edges of transparent objects more accurately. The design of this module helps to improve the boundary recognition effect of transparent medical waste and provide a reliable basis for subsequent grasping point detection.

[0193] As a possible implementation of this embodiment, the loss function of the multi-branch convolutional neural network model includes edge detection loss, coordinate positioning loss, and grasping point classification loss, as follows:

[0194]

[0195] Where: represents the edge detection loss; Indicates coordinate positioning loss; represents the classification loss of the grasping point; λ, γ, α, β all represent weight hyperparameters;

[0196]

[0197] Where: N represents the number of edge pixels; Represents the true edge label of the i-th pixel. If it belongs to the edge side, Otherwise, 0; Represents the edge probability value of the i-th pixel predicted by the model, with a value between [0,1];

[0198]

[0199] Where: M represents the number of grasping points; represents the coordinates of the jth real grasping point; represents the coordinates of the j-th grasp point predicted by the model;

[0200]

[0201] Where: represents the true label of the jth grasping point. If it is a graspable point, Otherwise, 0; It represents the graspability probability of the j-th grasp point predicted by the model, and its value is in the range of [0,1].

[0202] In this embodiment, the loss function includes edge detection loss, coordinate positioning loss and grasping point classification loss, which can optimize the model performance from multiple angles. Among them, the edge detection loss focuses on the edge extraction of transparent objects, the coordinate positioning loss ensures the precise position of the grasping point, and the grasping point classification loss ensures that the selected grasping point has high stability. Through the design of multimodal feature fusion, deep attention mechanism and comprehensive loss function, the edge detection and grasping point selection of transparent medical waste are more accurate and reliable, providing an effective technical solution for the visual grasping of transparent objects, and significantly improving the success rate and stability of robot automated processing of transparent objects.

[0203] As a possible implementation of this embodiment, step 4 includes the following steps:

[0204] Based on the trained multi-branch convolutional neural network model, a binary probability map is output, in which the value of each image represents the probability that the pixel belongs to the boundary;

[0205] A boundary detection threshold is set. If the predicted probability of a pixel is greater than or equal to the boundary detection threshold, the pixel is identified as a boundary point. Non-maximum suppression is used to eliminate redundant boundary detection results to ensure accurate and non-redundant edge detection.

[0206] Based on the multi-branch convolutional neural network model, the grasping point coordinates and graspability probability of the grasping point are output, and a classification threshold of the grasping point is preset. If the graspability probability of a grasping point is greater than or equal to the classification threshold, the point is considered to be a stable grasping point.

[0207] If multiple grab points meet the grab conditions, the best point is selected based on the following rules:

[0208] Maximize the distance to the boundary: Prioritize grasping points that are farther away from the edge of the object to avoid grasping failures due to boundary errors.

[0209] The center of gravity of the grasping point is close to the center of the target: Grasping points near the center of gravity of the object are preferred to improve the stability of the grasping.

[0210] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A deep learning visual capture method for transparent medical waste, characterized in that: The following steps are involved: Step 1: Collect multimodal data based on RGB, depth camera and infrared camera; Step 2: align the images captured by the RGB, depth camera and external infrared camera to generate three-channel composite image data; and generate different simulation data through data enhancement data; Step 3: A multi-branch convolutional neural network model is used. Each branch extracts feature information of different modalities on the shallow network. The deep network further optimizes the recognition effect of transparent object edges through the attention mechanism, and introduces a transparent object edge detection module to assist in edge positioning of transparent objects. Step 4: Use the trained multi-branch convolutional neural network model to detect the boundaries and appropriate grasping points of transparent medical waste, and the robot automatically selects the most stable grasping point based on the preset threshold.

2. The deep learning visual capture method for transparent waste according to claim 1, characterized in that: The specific steps of registering the images collected by the RGB, depth camera and external infrared camera are as follows: Feature extraction: SIFT feature detection algorithm is used to extract key feature points from RGB, depth and infrared images to obtain feature points and feature vectors in each image; Feature point matching: Using Euclidean distance judgment, feature point matching is performed between RGB images and other modality images to find point pairs with similar features: Where: f R represents the feature vector extracted from the RGB image; f M represents a feature vector found in other modality images, where the other modality images are depth camera images or infrared camera images; If the Euclidean distance between two point pairs is less than the preset threshold, the two point pairs are considered to be feature point pairs; Registration-affine transformation: Based on the matched feature point pairs, an affine transformation is established to convert the feature points in the RGB image into the coordinate system of the depth or infrared image: Where: (x R ,y R ) represents the coordinates of the feature points in the RGB image; (x M ,y M ) represents the coordinates of the corresponding feature points in the depth or infrared image; a, b, c, d represent scaling and rotation parameters; (t x ,t y ) represents the translation parameter; Solve the radial transformation parameters: Solve the affine transformation parameters by the least squares method to minimize the error. After N pairs of feature point registration, establish the following minimization problem: Taking the derivative of the above minimization problem and setting the derivative to zero, we get the following linear equations: By solving the linear equations, we can get the radiation transformation parameters a, b, c, d, t x ,t y , complete the registration; Generate three-channel composite image data: Combine the registered RGB, depth, and infrared images into a three-channel composite image.

3. The deep learning visual capture method for transparent medical waste according to claim 2, characterized in that: The specific steps of feature extraction are as follows: Image preprocessing: Perform multi-resolution scaling on RGB, depth, and infrared images to construct images at multiple resolution levels; convert the scaled RGB images into grayscale images while retaining the grayscale characteristics of the depth image and infrared image. The grayscale conversion formula is as follows: I gray =0.2989×I R +0.5870×I G +0.1140×I B ; Where: I gray Represents a grayscale image; I R ,I G ,I B Represents the red, green, and blue channels of an RGB image; Construct a multi-scale Gaussian pyramid: For each level of resolution image I r Apply Gaussian blur to form a multi-scale space; generate a series of Gaussian blurred images through different Gaussian blur parameters: Where: G(x, y, σ) represents the Gaussian blurred image; x, y represent the pixel positions; σ represents the blur scale parameter, which determines the blur intensity; e represents a mathematical constant or a natural constant; A set of images with different blur scales are generated on each level of resolution to form a pyramid structure. Assuming that the image in the Gaussian scale space is L(x, y, σ), the scale pyramid of each level is expressed as: L(x,y,σ)=G(x,y,σ)*I r ; Where: L(x, y, σ) represents images at different scales; * represents convolution operation; Construct a differential Gaussian pyramid: In each level of the Gaussian pyramid, adjacent fuzzy scale images are differentiated to construct a differential Gaussian pyramid to detect significant feature areas in the image: D(x,y,σ)=L(x,y,kσ)-L(x,y,σ); Where: D(x, y, σ) represents the differential Gaussian image; k represents the scale factor, which is used to generate images of different scales; In the differential Gaussian image of each level of resolution, a 3×3×3 neighborhood extreme value detection is performed on each pixel, and the maximum or minimum value points are detected in the space and scale domains, and the maximum or minimum value points are marked as candidate key points; Multi-resolution key point positioning and filtering: Accurately locate multi-resolution candidate key points, remove low-contrast feature points and unstable edge points, and optimize the key point position through Taylor expansion: Where: Represents the gradient at the key point; represents the Hessian matrix; Determine the edge response value based on the Hessian matrix, filter out edge points, and ensure that the key point set is concentrated in the stable area; Direction assignment: Gradient assignment and direction are calculated within the neighborhood of each multi-resolution keypoint: Where: m(x,y) represents the gradient assignment; θ(x,y) represents the direction angle; Construct a direction histogram for the neighborhood of each key point, select the main direction and assign it to the key point to achieve rotation invariance; Feature vector generation and multi-resolution fusion: For each key point of each resolution, a 128-dimensional feature vector is generated, the neighborhood is divided into blocks, and the weighted gradients in 8 directions are calculated in each block to obtain the feature descriptor; The feature points and descriptors of all resolution levels are fused by weighted averaging, where the weights are assigned according to the importance of each resolution level; The feature vector obtained by normalizing the feature fusion is used to obtain the final extracted feature points and the feature vectors corresponding to the feature points.

4. The deep learning visual capture method for transparent medical waste according to claim 3 is characterized in that: The specific steps of fusing the feature points and descriptors of all resolution levels by weighted averaging are as follows: Where: f(i) represents the importance function of resolution i to the feature descriptor; w i represents the weight of the i-th resolution level; d i represents the feature descriptor of the i-th resolution level; L represents the total number of resolution levels; Where: SNR i represents the signal-to-noise ratio of the i-th resolution, evaluated by contrast and noise level; σ i Represents the standard deviation of the i-th resolution level.

5. The deep learning visual capture method for transparent medical waste according to claim 2, characterized in that: The generating of different simulation data includes performing data enhancement on the generated three-channel composite image to generate simulation data, wherein the data enhancement includes: performing random angle rotation on the composite image, scaling the composite image proportionally, performing random translation on the composite image, adding Gaussian noise to the composite image, or performing mirror flipping on the composite image, or a combination of these.

6. The deep learning visual capture method for transparent medical waste according to claim 1, characterized in that: The multi-branch convolutional neural network model includes an input layer, a shallow branch feature extraction network, a deep attention mechanism network and a transparent object edge detection module; Input layer: used to input three-channel composite image data, including RGB, depth and infrared channels. The input image size is H×W×3, where H represents height, W represents width, and 3 represents 3 channels. The shallow branch feature extraction network includes RGB image branch, depth image branch and infrared image branch. The three branches are independent of each other and are used to extract feature information of RGB, depth and infrared images respectively. Each branch shares the convolutional network structure, but the parameters are not shared, as follows: RGB branch: Input: RGB channel data; Convolutional layer 1: The convolution kernel size is 3×3, the stride is 1, the number of output channels is 64, and the feature map of H×W×64 is obtained; Pooling layer 1: It is the maximum pooling layer with a window size of 2×2, a stride of 2, and an output size of The feature map of Convolutional layer 2: The convolution kernel size is 3×3, the stride is 1, and the number of output channels is 128. The feature map of Pooling layer 2: It is the maximum pooling layer with a window size of 2×2, a stride of 2, and an output size of The feature map of Depth image branch: Input: depth channel data; The structure is the same as the RGB branch, and the convolution and pooling operations are performed continuously to obtain the output size The feature map of Infrared imaging branch: Input: infrared channel data; The structure is the same as the RGB branch, and the convolution and pooling operations are performed continuously to obtain the output size The feature map of Shallow branch output: The outputs of the three branches are concatenated in the channel dimension to obtain Composite feature map of Deep Attention Mechanism Network: Convolutional layer: The input is a composite feature map, the convolution kernel size is 3×3, the stride is 1, the number of output channels is 256, and the output is The feature map of Attention mechanism module: Input: feature map output by the convolutional layer; Channel attention: weight each channel and generate a weighted feature map; Transparent object edge detection module: including edge detection convolution and grasping point detection output layer; The edge detection convolution includes: Convolutional layer: The input is the output feature of the deep attention mechanism, the convolution kernel size is 3×3, the stride is 1, the number of output channels is 128, the activation function uses the ReLU activation function, and the output size is Grasp point detection output layer: Input: feature map after edge detection; Convolution kernel size: 1×1, stride 1, number of output channels 2; output size is That is, the probability map of the grasping point position, which is the probability distribution of x and y coordinates respectively; Softmax activation: The Softmax activation function is used on the grasping point position probability map to obtain the probability distribution, which is used to finally select the most appropriate grasping point.

7. The deep learning visual capture method for transparent medical waste according to claim 6, characterized in that: The loss function of the multi-branch convolutional neural network model includes edge detection loss, coordinate positioning loss, and grasping point classification loss, as follows: Where: represents the edge detection loss; Indicates coordinate positioning loss; represents the grasp point classification loss; λ, γ, α, and β all represent weight hyperparameters; Where: N represents the number of edge pixels; Represents the true edge label of the i-th pixel. If it belongs to the edge side, Otherwise, 0; Represents the edge probability value of the i-th pixel predicted by the model, with a value between [0,1]; Where: M represents the number of grasping points; represents the coordinates of the jth real grasping point; represents the coordinates of the j-th grasp point predicted by the model; Where: represents the true label of the jth grasping point. If it is a graspable point, Otherwise, 0; It represents the graspability probability of the j-th grasp point predicted by the model, and its value is in the range of [0,1].

Images