A method for transparent object recognition based on RGB-D feature fusion enhancement network

Abstract

This invention relates to the field of image processing technology, and provides a method for transparent object recognition based on an RGB-D feature fusion enhancement network. The method includes: acquiring an RGB-D image of the transparent object, and preprocessing the image by performing RGB and depth estimation to obtain a depth image D2, resulting in an RGB-D2 bimodal aligned image; constructing an improved YOLOv8-seg network: designing a C2f_ScConv module to replace the C2f module in the original network backbone; designing a global hybrid feature attention enhancement module (HAM) and inserting it after the SPPF module in the original network backbone; and designing a multi-dimensional feature fusion module (FAM) to replace the Concat module in the original network neck. This invention effectively suppresses background redundancy information irrelevant to the transparent object recognition task, while enhancing weak texture information such as edges and contours of the transparent foreground, thus improving the accuracy of model recognition.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of image processing technology, and more particularly to a transparent object recognition method based on an RGB-D feature fusion enhancement network. Background Technology

[0002] Transparent objects (such as glassware, optical lenses, and transparent packaging) are widely used in daily life, industrial production, and laboratory settings due to their unique optical properties (transmittance, refraction, and low texture) and aesthetic appeal. Examples include: in the medical field, automated operation of experimental equipment such as transparent petri dishes and test tubes; in consumer electronics, precision assembly of mobile phone glass screens and camera lenses; and in logistics sorting, automated gripping of transparent plastic bottles or packaging boxes.

[0003] However, the physical characteristics of transparent objects lead to significant differences from regular objects, specifically manifested in visual perception challenges due to optical properties, including light transmission, complex reflections, and refraction interference, which traditional machine vision recognition methods struggle to handle effectively. Therefore, for certain special objects such as transparent ones, machine vision still faces difficulties such as low recognition accuracy or long processing times, easily leading to failures in downstream tasks like grasping. Thus, there is an urgent need for a transparent object recognition solution that is compatible with varying lighting conditions, complex backgrounds, and multi-object interaction scenarios, while providing high-precision input for grasping control. Summary of the Invention

[0004] To address the above issues, this invention replaces each C2f module in the original network with a designed C2f_ScConv module. To address the inability to accurately identify the contours and boundaries of transparent objects using only RGB information, Depth Anything V2 is employed for depth estimation, and a globally hybrid feature attention enhancement module is designed to further enhance the weak texture features of transparent objects by combining depth features with RGB features. Furthermore, to address the limitation of existing methods that rely solely on simple channel concatenation during feature fusion, which fails to efficiently fuse detailed and semantic information, a multi-dimensional feature fusion module is designed to replace each Concat operation in the original network's neck section.

[0005] According to an embodiment of the present invention, a method for transparent object recognition based on an RGB-D feature fusion enhancement network is provided.

[0006] In a first aspect of the invention, a method for transparent object recognition based on an RGB-D feature fusion enhancement network is provided. The method includes:

[0007] Step S01: Acquire the RGB-D image of the transparent object, and preprocess the RGB and depth images D2 obtained by depth estimation to obtain the RGB-D2 bimodal aligned image;

[0008] Step S02: Construct an improved YOLOV8-seg network: Design the C2f_ScConv module to replace the C2f module in the backbone of the original network; design the global hybrid feature attention enhancement module HAM and insert it after the SPPF module in the backbone of the original network; design the multi-dimensional feature fusion module FAM to replace the Concat module in the neck of the original network.

[0009] Further, the C2f_ScConv module mentioned in step S02 includes a 1×1 convolution, two BN layers, two SiLU activation functions, and a ScConv module; the input image RGB is sequentially passed through a 1×1 convolution, a BN layer, and a SiLU activation function, and then split into two input feature maps in half according to the channel dimension. The output feature map obtained by passing one of the input feature maps through the ScConv module is concatenated with the other input feature map in the channel dimension. The concatenated feature map is then passed through a BN layer and a SiLU activation function to obtain the output of the C2f_ScConv module.

[0010] Furthermore, the ScConv module includes a spatial dimension reconstruction convolution module and a channel dimension reconstruction convolution module.

[0011] Furthermore, the specific content of the spatial dimension reconstruction convolution module is as follows:

[0012] Step S021S: Perform GroupBatchnorm2d normalization on the input feature map to obtain a normalized feature map;

[0013] Step S022S: Apply the softmax activation function to the trainable parameter part of the normalized feature map to obtain the gating coefficients;

[0014] Step S023S: Multiply the normalized feature map and the gating coefficients pixel by pixel and pass them through a sigmoid activation function to obtain the gating weights;

[0015] Step S024S: The part with the gate coefficient greater than the threshold is the information mask, and the part with the gate coefficient less than the threshold is the non-information mask. The information mask is multiplied pixel by pixel with the gate weight and then added pixel by pixel with the input feature map to obtain the weighted feature of the information part.

[0016] Step S025S: Multiply the non-information mask and the gate weights pixel by pixel and then concatenate them with the input feature map to obtain the weighted features of the non-information part;

[0017] Step S026S: After splitting the weighted features of the information part in half according to the channel dimension, we obtain the features of the first information part and the features of the second information part. After splitting the weighted features of the non-information part in half according to the channel dimension, we obtain the features of the first non-information part and the features of the second non-information part.

[0018] Step S027S: The output of the pixel-by-pixel summation of the features of the first information part and the features of the second non-information part is concatenated with the output of the pixel-by-pixel summation of the features of the second information part and the features of the first non-information part according to the channel dimension to obtain the output of the spatial dimension reconstruction convolution module.

[0019] Furthermore, the content of the channel dimension reconstruction convolution module is as follows:

[0020] Step S021C: Take the output of the spatial dimension reconstruction convolutional module as input and split it into two parts, up feature map and low feature map, according to the number of channels of the output of the spatial dimension reconstruction convolutional module;

[0021] Step S022C: After performing channel compression on the up feature map, the up feature map is divided into two parts. One part is processed by a 3×3 depthwise DW convolution, and the other part is processed by a 1×1 pointwise PW convolution. Then, pixel-wise addition is performed to obtain the up_refined feature map.

[0022] Step S023C: After performing channel compression on the low feature map, the output after a 1×1 pointwise PW convolution is concatenated with the low feature map along the channel dimension to obtain the low_fused feature map.

[0023] Step S024C: Concatenate the up_refined feature map and the low_fused feature map along the channel dimension to obtain the concatenated feature map. Multiply the output after adaptive average pooling and softmax activation function with the concatenated feature map pixel by pixel to obtain the output feature map.

[0024] Step S025C: Split the output feature map into two parts according to the number of channels, and add the two parts pixel by pixel to obtain the output of the final channel dimension reconstruction convolution module.

[0025] Furthermore, the specific content of HAM mentioned in step S02 is as follows:

[0026] Step S021H: After passing the depth image D2 through a 3×3 convolution, a BN layer, a GELU activation function, a max pooling layer, a 3×3 convolution, a BN layer, and a GELU activation function in sequence, the output is bilinearly interpolated and aligned with the RGB features in the resolution dimension to obtain the depth prior features;

[0027] Step S022H: The RGB features are passed through a 1×1 convolution to align with the depth prior features in the channel dimension. After being concatenated with the depth prior features in the channel dimension, they are passed through a 3×3 convolution, a BN layer, and a GELU activation function to obtain the shallow fusion features.

[0028] Step S023H: Adjust the dimensional order of the shallow fusion features: [batch size b, channel c, height H, width W] to: [batch size b, channel c, h*window_size, w*window_size], and then rearrange them to: [batch size b*h*w, window_size*window_size, channel c];

[0029] Step S024H: The rearranged shallow fusion features are processed through a multi-head self-attention mechanism to obtain a self-attention map, and the self-attention map is rearranged from the dimensions of [batch size b*h*w, window_size*window_size, channel c] to: [batch size b, channel c, height H, width W];

[0030] Step S025H: Multiply the rearranged self-attention map with the RGB features pixel by pixel, and then add it to the RGB features pixel by pixel to obtain the final output.

[0031] Furthermore, the specific content of FAM mentioned in step S02 is as follows:

[0032] Step S021F: After passing through the channel dimension enhancement module, the shallow feature map is multiplied pixel by pixel with the shallow feature map to obtain the channel dimension enhancement coefficient map of the shallow feature map.

[0033] Step S022F: After the deep feature map passes through the channel dimension enhancement module, it is multiplied pixel by pixel with the deep feature map to obtain the channel dimension enhancement coefficient map of the deep feature map;

[0034] Step S023F: After passing the channel dimension enhancement coefficient map of the shallow feature map through the spatial dimension enhancement module, multiply it pixel by pixel with the channel dimension enhancement coefficient map of the shallow feature map to obtain the spatial dimension enhancement coefficient map of the shallow feature map.

[0035] Step S024F: After passing the channel dimension enhancement coefficient map of the deep feature map through the spatial dimension enhancement module, multiply it pixel by pixel with the channel dimension enhancement coefficient map of the deep feature map to obtain the spatial dimension enhancement coefficient map of the deep feature map.

[0036] Step S025F: Finally, the spatial dimension enhancement coefficient map of the shallow feature map and the spatial dimension enhancement coefficient map of the deep feature map are concatenated along the channel dimension to obtain the final output of the model.

[0037] Furthermore, the channel dimension enhancement module comprises: dividing the input feature map into two parts; one part undergoes adaptive average pooling, channel compression, and dimension order adjustment to obtain an average feature map; the other part undergoes adaptive max pooling, channel compression, and dimension order adjustment to obtain a global average feature map; the average feature map and the global average feature map are concatenated along the channel dimension, and then subjected to a 5×5 convolution and a sigmoid activation function to obtain a channel dimension enhancement coefficient map.

[0038] Furthermore, the spatial dimension enhancement module is as follows: taking the output feature map of the channel dimension enhancement module as input, the input feature map is divided into two parts, one part is subjected to average pooling, and the other part is subjected to max pooling. After concatenating the output of average pooling and the output of max pooling in the channel dimension, the result is obtained by passing a 5×5 convolution and a sigmoid activation function to obtain the channel dimension enhancement coefficient map.

[0039] In a second aspect of the invention, an apparatus for transparent object recognition based on an RGB-D feature fusion enhancement network is provided. The apparatus includes:

[0040] Image acquisition module: used to acquire RGB-D images of transparent objects, and preprocess the depth image D2 obtained by RGB and depth estimation of the image to obtain an RGB-D2 bimodal aligned image;

[0041] Model building module: used to build an improved YOLOV8-seg network: design the C2f_ScConv module to replace the C2f module in the backbone of the original network; design the global hybrid feature attention enhancement module HAM and insert it after the SPPF module in the backbone of the original network; design the multi-dimensional feature fusion module FAM to replace the Concat module in the neck of the original network.

[0042] In a third aspect of the invention, an electronic device is provided. The electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the program to implement the method according to a first aspect of the invention.

[0043] In a fourth aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to a first aspect of the invention.

[0044] This invention replaces each C2f module in the original network with a designed C2f_ScConv module. Addressing the issue that RGB information alone cannot accurately identify the contours and boundaries of transparent objects, it employs Depth Anything V2 for depth estimation and uses a globally hybrid feature attention enhancement module to further enhance the weak texture features of transparent objects by combining depth features with RGB features. Furthermore, recognizing that existing methods rely solely on simple channel concatenation during feature fusion, which fails to efficiently fuse detailed and semantic information, this invention replaces each Concat operation in the original network's neck section with a multi-dimensional feature fusion module.

[0045] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description.

[0046] The beneficial effects of this invention are:

[0047] 1. The present invention designs a C2f_ScConv module to replace each C2f module in the original network, which can effectively suppress background redundant information that is irrelevant to the transparent object recognition task, while strengthening weak texture information such as edges and contours of transparent foregrounds, and further improving the accuracy of model recognition;

[0048] 2. This invention designs a global hybrid feature attention enhancement module (HAM), and uses the depth map after depth estimation by the Depth Anything V2 model and the original RGB image as input to the module. By combining geometric information including transparent foreground expansion, the model's ability to distinguish between foreground and background, especially boundaries, can be improved.

[0049] 3. The present invention designs a multi-dimensional feature fusion module FAM to replace each Concat operation in the original network neck part, which can effectively fuse shallow detailed texture information with deep abstract semantic information, and help improve the model's fitting ability. Attached Figure Description

[0050] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. Wherein:

[0051] Figure 1A flowchart of a method for transparent object recognition based on an RGB-D feature fusion enhancement network according to an embodiment of the present invention is shown;

[0052] Figure 2 A network structure diagram of an RGB-D feature fusion enhancement network based on an embodiment of the present invention is shown;

[0053] Figure 3 A structural diagram of the C2f_ScConv module according to an embodiment of the present invention is shown;

[0054] Figure 4 A structural diagram of a HAM module according to an embodiment of the present invention is shown;

[0055] Figure 5 A structural diagram of the FAM module according to an embodiment of the present invention is shown;

[0056] Figure 6 A visual comparison diagram of the recognition capabilities of the original YOLOV8-seg network according to an embodiment of the present invention is shown;

[0057] Figure 7 A block diagram of a device for transparent object recognition based on an RGB-D feature fusion enhancement network according to an embodiment of the present invention is shown;

[0058] Figure 8 A schematic diagram of a device for transparent object recognition based on an RGB-D feature fusion enhancement network according to an embodiment of the present invention is shown. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] According to an embodiment of the present invention, a method for transparent object recognition based on an RGB-D feature fusion enhancement network is proposed. By designing a C2f_ScConv module to replace each C2f module in the original network, and addressing the issue that the contours and boundaries of transparent objects cannot be accurately identified using only RGB information, Depth Anything V2 is used for depth estimation, and a globally hybrid feature attention enhancement module is designed to further enhance the weak texture features of transparent objects by combining depth features with RGB features. Furthermore, addressing the issue that existing methods rely solely on simple channel concatenation during feature fusion, which fails to efficiently fuse detailed information with semantic information, a multi-dimensional feature fusion module is designed to replace each Concat operation in the neck part of the original network.

[0061] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0062] Figure 1 This is a schematic flowchart of a method for transparent object recognition based on an RGB-D feature fusion enhancement network according to an embodiment of the present invention. The method includes:

[0063] Step S01: Acquire the RGB-D image of the transparent object, and preprocess the RGB and depth images D2 obtained by depth estimation to obtain the RGB-D2 bimodal aligned image;

[0064] Step S02: Construct an improved YOLOV8-seg network: Design the C2f_ScConv module to replace the C2f module in the backbone of the original network; design the global hybrid feature attention enhancement module HAM and insert it after the SPPF module in the backbone of the original network; design the multi-dimensional feature fusion module FAM to replace the Concat module in the neck of the original network.

[0065] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0066] To provide a clearer explanation of the above-described method for transparent object recognition based on RGB-D feature fusion enhancement networks, a specific embodiment will be used for illustration below. However, it is worth noting that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation of the present invention.

[0067] The following concrete example will further illustrate the method for transparent object recognition based on RGB-D feature fusion enhancement networks:

[0068] Step S01: Acquire the RGB-D image of the transparent object, and preprocess the RGB and depth images D2 obtained by depth estimation to obtain the RGB-D2 bimodal aligned image.

[0069] In this embodiment, the specific steps of step S01 are as follows:

[0070] Step S011: Use the depth camera to acquire the RGB-D image of the current transparent object scene, align the RGB stream with the D stream, and save the RGB-D image to the local machine.

[0071] Preferably, the depth camera uses Intel's RealSense D435i, with the RGB stream resolution set to 640×480, BGR8 channel format, 30fps, and the D stream resolution set to 640×480, 16-bit depth channel format, 30fps. The RGB stream is saved in uint8 data format, and the D stream is saved in uint16 data format.

[0072] Step S012: Input the raw RGB image captured by the camera into the Depth Anything V2 depth estimation model to obtain the normalized relative depth map D2 of the transparent object. Using D as a reference, map D2 from the float32 data format to the uint16 data format of D to calibrate the absolute depth range of D2.

[0073] Step S02: Construct an improved YOLOV8-seg network. Design a C2f_ScConv module to replace the original C2f module in the backbone part of the network. Use RGB-D as the input of the network backbone part to suppress redundant information unrelated to the foreground and enhance the weak texture feature information of transparent objects. Design a global hybrid feature attention enhancement module and insert it after the SPPF module in the original network backbone part to enhance the model's attention to the edge and boundary region feature information of transparent objects. Design a multi-dimensional feature fusion module to replace the Concat module in the original network neck part to efficiently fuse the detailed information of shallow feature maps from multiple stages with the semantic information of deep features.

[0074] like Figure 2The diagram shown is a network structure diagram of the improved YOLOV8-seg network of the present invention. The acquired RGB image is passed through the backbone, neck and head parts to obtain the final recognition result. First, the RGB image passes through the CBS, CBS, C2f_ScConv, CBS, C2f_ScConv, CBS, C2f_ScConv, CBS, C2f_ScConv, SPPF module and HAM module of the backbone part in sequence to obtain the feature network extracted by the backbone part.

[0075] like Figure 3 As shown, the C2f_ScConv module includes a 1×1 convolution, two Batch Normalization (BN) layers, two SiLU activation functions, and a ScConv module. First, the input image (RGB) is sequentially passed through a 1×1 convolution, a Batch Normalization (BN) layer, and a SiLU activation function. Then, the output is split in half along the channel dimension, representing input feature map x1 and input feature map x2. The output of input feature map x2 after passing through the ScConv module is concatenated with input feature map x1 along the channel dimension, and then passed through another 1×1 convolution, BN layer, and SiLU activation function to obtain the output `out` of the C2f_ScConv module. The entire process can be represented as:

[0076]

[0077] In a preferred embodiment of the present invention, the ScConv module includes a spatial dimension reconstruction convolution module (SRU) and a channel dimension reconstruction convolution module (CRU).

[0078] like Figure 3As shown, the SRU module first performs GroupBatchnorm2d normalization on the input feature map x2 to obtain gn_x; then, the trainable parameters of gn_x are activated by a softmax function to obtain the gating coefficients w_gamma; next, gn_x and w_gamma are multiplied pixel by pixel and passed through a sigmoid activation function to obtain the gating weights reweigts; then, based on w_gamma and a preset threshold, an information mask info_mask and a non-information mask noninfo_mask are generated, where the portion of w_gamma greater than the threshold is info_mask and the portion less than the threshold is noninfo_mask; subsequently, the info_mask is... After multiplying _mask and reweigts pixel-by-pixel, it is added to the input feature map x2 pixel-by-pixel to obtain the weighted feature x_1 of the information part. After multiplying noninfo_mask and reweigts pixel-by-pixel, it is concatenated with the input feature map x2 to obtain the weighted feature x_2 of the non-information part. Finally, x_1 is split in half according to the channel dimension to obtain x_11 and x_12, and x_2 is split in half according to the channel dimension to obtain x_21 and x_22. The output of the pixel-by-pixel addition of x_11 and x_22 is concatenated with the output of the pixel-by-pixel addition of x_12 and x_21 according to the channel dimension to obtain the final output SRU_out of the SRU module. Preferably, the preset threshold is set to 0.5. The entire process can be represented as:

[0079]

[0080] like Figure 3As shown, the CRU module first takes SRU_out as input and splits it into two parts based on the number of channels in SRU_out, represented as an up feature map and a low feature map. The number of channels in the up feature map is alpha * the original number of channels in SRU_out, and the number of channels in the low feature map is the original number of channels minus the number of channels in the up feature map. After channel compression, the up feature map is divided into two parts. One part is processed by a 3×3 depthwise DW convolution, and the other part is processed by a 1×1 pointwise PW convolution, followed by pixel-wise summation to obtain up_r. The refined feature map Y1 is processed by channel compression of the low feature map, followed by a 1×1 pointwise PW convolution. The output is then concatenated with the low feature map along the channel dimension to obtain the up-refined feature map Y2. Y1 and Y2 are then concatenated along the channel dimension to form Y. The output of Y, after adaptive average pooling and softmax activation, is multiplied pixel-wise with Y, resulting in the output out. Finally, out is split into two parts according to the number of channels, and the two parts are added pixel-wise to obtain the final output of the CRU module. The entire process can be represented as:

[0081]

[0082] like Figure 4As shown, the Global Hybrid Feature Attention Enhancement (HAM) module takes RGB features and a depth image D2 as input. First, D2 is sequentially processed through a 3×3 convolution, a batch normalization (BN) layer, a GELU activation function, a max pooling layer, another 3×3 convolution, another BN layer, and another GELU activation function. The output is then bilinearly interpolated to align with the RGB features in the resolution dimension, resulting in the depth prior feature `depth_feat`. Next, the RGB features are processed through a 1×1 convolution to align with `depth_feat` in the channel dimension. After concatenation with `depth_feat` in the channel dimension, the concatenation is then processed through a 3×3 convolution, a BN layer, and another GELU activation function to obtain the shallow fusion feature `fused_feat`. The dimensions of `fused_feat` are arranged in the following order: [batch size...]. The parameters [b, channel c, height H, width W] are adjusted to [b, c, h*window_size, w*window_size], and then rearranged to [b*h*w, window_size*window_size, c]. The rearranged fused_feat is then processed by the MultiheadAttention mechanism in the Transformer to obtain the attention map attn_out. This attn_out is then rearranged from the dimensions [b*h*w, window_size*window_size, c] to [b, c, H, W]. Finally, the rearranged attn_out is multiplied pixel-by-pixel with the RGB features and then added pixel-by-pixel to obtain the final output. The entire process can be represented as:

[0083]

[0084] Then, the output feature map of the HAM module is used as the input of the neck part, and passes through the following modules in sequence: Upsample, FAM, C2f, Upsample, FAM, C2f, CBS, FAM, C2f, CBS, FAM, C2f. The feature map output by the HAM module in the backbone part is used as another input of the fourth FAM module, and the feature map output by the first C2f module is used as another input of the third FAM module, finally obtaining the fused feature map of the neck part.

[0085] like Figure 5As shown, the multi-dimensional feature fusion module FAM takes a shallow feature map x1 and a deep feature map x2 as input. x1 is processed through the channel dimension enhancement module (Channel_Attention) and then multiplied pixel-by-pixel with the input x1 to obtain the channel dimension enhancement coefficient map x1_channel. Similarly, x2 is processed through the channel dimension enhancement module and then multiplied pixel-by-pixel with the input x2 to obtain the channel dimension enhancement coefficient map x2_channel. x1_channel is processed through the spatial dimension enhancement module (Spatial_Attention) and then multiplied pixel-by-pixel with the input x1_channel to obtain the spatial dimension enhancement coefficient map x1_spatial. The same process is repeated for x2_channel. Finally, x1_spatial and x2_spatial are concatenated along the channel dimension to obtain the final output of the model. The entire process can be represented as:

[0086]

[0087] like Figure 5 As shown, the channel dimension enhancement module divides the input feature map into two parts. One part undergoes adaptive average pooling, channel compression, and dimension order adjustment to obtain the average feature map avg; the other part undergoes adaptive max pooling, channel compression, and dimension order adjustment to obtain the global average feature map max_. avg and max_ are concatenated along the channel dimension, and then passed through a 5×5 convolution and a sigmoid activation function to obtain the channel dimension enhancement coefficient map. The entire process can be represented as:

[0088]

[0089] like Figure 5 As shown, the spatial dimension enhancement module takes the output feature map of the channel dimension enhancement module as input, divides the input feature map into two parts, one part is subjected to average pooling, and the other part is subjected to max pooling. The outputs of the average pooling and max pooling are then concatenated along the channel dimension, followed by a 5×5 convolution and a sigmoid activation function to obtain the channel dimension enhancement coefficient map. The entire process can be represented as:

[0090] out = sigmoid(Conv) 7×7 (concat[avg(x),max(x)]))

[0091] Finally, the feature maps output by the second, third, and fourth C2f in the neck part are used as input to the head part, and the final recognition result is output.

[0092] The effectiveness of this invention will be further illustrated below with simulation experiments.

[0093] 1. Experimental environment configuration and parameter settings:

[0094] The operating system used in the experiment was Ubuntu 20.04, the CPU was a 15v CPU Intel(R) Xeon(R) Platinum 8474C, the graphics card was one RTX 4090D (24GB), and the memory was 24GB.

[0095] The training parameters of the model are as follows: the weights are initialized using the official open-source weights of YOLOV8s.pt, the batch size is set to 16, the resolution of the input image is 640×640, the number of training epochs is set to 100, the initial learning rate is set to 0.01, SGD is selected as the optimizer, and the momentum is set to 0.937.

[0096] 2. Experiment Content:

[0097] This invention trains a network model using the TROSD dataset proposed by the Tsinghua University team and conducts experimental comparisons and verifications with existing methods. The comparison methods include the original YOLOV8-seg network, BiseNetV2 in author Yu C's "Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation", and PSPNet in author Zhao H's "Pyramid scene parsing network".

[0098] This invention employs some commonly used evaluation criteria in deep repair tasks, including Intersection over Union (IoU) and Mean Intersection over Union (mIoU), with the specific calculation formulas as follows:

[0099]

[0100] Where IoU represents the ratio of the intersection to the union of the recognition results of all categories with the ground truth (GT), TP represents the number of correctly predicted samples, FP represents the number of incorrectly predicted samples, FN represents the number of samples that are predicted as other categories, and mIoU represents the average of the IoU values ​​of all categories after summing.

[0101] Table 1

[0102]

[0103] Based on the experimental comparison results in Table 1, it can be seen that all the indicators of the present invention have obtained the best quantitative results. Figure 6 The visualization and qualitative comparison between the present invention and the original YOLOv8s-seg network shows that the present invention can better segment the outline, boundary and whole of transparent objects.

[0104] Based on the same inventive concept, this invention also proposes a device for transparent object recognition based on an RGB-D feature fusion enhancement network. The implementation of this device can be found in the implementation of the method described above; repeated details will not be repeated. Figure 7 As shown, the device 100 includes:

[0105] Image acquisition module 101: used to acquire RGB-D images of transparent objects, and preprocess the depth image D2 obtained by RGB and depth estimation of the image to obtain an RGB-D2 bimodal aligned image;

[0106] Model building module 102: Used to build an improved YOLOV8-seg network: Design the C2f_ScConv module to replace the C2f module in the backbone of the original network; design the global hybrid feature attention enhancement module HAM and insert it after the SPPF module in the backbone of the original network; design the multi-dimensional feature fusion module FAM to replace the Concat module in the neck of the original network.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0108] like Figure 8 As shown, the device includes a central processing unit (CPU), which can perform various appropriate actions and processes based on computer program instructions stored in read-only memory (ROM) or loaded from storage units into random access memory (RAM). The RAM can also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0109] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0110] The processing unit executes the various methods and processes described above, such as method steps S01 to S02. For example, in some embodiments, method steps S01 to S02 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of method steps S01 to S02 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute method steps S01 to S02 by any other suitable means (e.g., by means of firmware).

[0111] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0112] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0113] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0114] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0115] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for transparent object recognition based on an RGB-D feature fusion enhancement network, characterized in that, The method includes: Step S01: Acquire the RGB-D image of the transparent object, and preprocess the RGB and depth images D2 obtained by depth estimation to obtain the RGB-D2 bimodal aligned image; Step S02: Construct an improved YOLOV8-seg network: Design the C2f_ScConv module to replace the C2f module in the backbone of the original network; design the global hybrid feature attention enhancement module HAM and insert it after the SPPF module in the backbone of the original network; design the multi-dimensional feature fusion module FAM to replace the Concat module in the neck of the original network. The specific content of the HAM is as follows: Step S021H: After passing the depth image D2 through a 3×3 convolution, a BN layer, a GELU activation function, a max pooling layer, a 3×3 convolution, a BN layer, and a GELU activation function in sequence, the output is bilinearly interpolated and aligned with the RGB features in the resolution dimension to obtain the depth prior features; Step S022H: The RGB features are passed through a 1×1 convolution to align with the depth prior features in the channel dimension. After being concatenated with the depth prior features in the channel dimension, they are passed through a 3×3 convolution, a BN layer, and a GELU activation function to obtain the shallow fusion features. Step S023H: Adjust the dimensional order of the shallow fusion features: [batch size b, channel c, height H, width W] to: [batch size b, channel c, h*window_size, w*window_size], and then rearrange them to: [batch size b*h*w, window_size*window_size, channel c]; Step S024H: The rearranged shallow fusion features are processed through a multi-head self-attention mechanism to obtain a self-attention map, and the self-attention map is rearranged from the dimensions of [batch size b*h*w, window_size* window_size, channel c] to: [batch size b, channel c, height H, width W]. Step S025H: Multiply the rearranged self-attention map with the RGB features pixel by pixel, and then add it to the RGB features pixel by pixel to obtain the final output.

2. The method for transparent object recognition based on an RGB-D feature fusion enhancement network according to claim 1, characterized in that, The C2f_ScConv module mentioned in step S02 includes a 1×1 convolution, two BN layers, two SiLU activation functions, and a ScConv module. The input image RGB is sequentially passed through a 1×1 convolution, a BN layer, and a SiLU activation function, and then split into two input feature maps in half according to the channel dimension. The output feature map obtained by passing one of the input feature maps through the ScConv module is concatenated with the other input feature map in the channel dimension. The concatenated feature map is then passed through a BN layer and a SiLU activation function to obtain the output of the C2f_ScConv module.

3. The method for transparent object recognition based on an RGB-D feature fusion enhancement network according to claim 2, characterized in that, The ScConv module includes a spatial dimension reconstruction convolution module and a channel dimension reconstruction convolution module.

4. The method for transparent object recognition based on an RGB-D feature fusion enhancement network according to claim 3, characterized in that, The specific content of the spatial dimension reconstruction convolution module is as follows: Step S021S: Perform GroupBatchnorm2d normalization on the input feature map to obtain a normalized feature map; Step S022S: Apply the softmax activation function to the trainable parameter part of the normalized feature map to obtain the gating coefficients; Step S023S: Multiply the normalized feature map and the gating coefficients pixel by pixel and pass them through a sigmoid activation function to obtain the gating weights; Step S024S: The part with the gate coefficient greater than the threshold is the information mask, and the part with the gate coefficient less than the threshold is the non-information mask. The information mask is multiplied pixel by pixel with the gate weight and then added pixel by pixel with the input feature map to obtain the weighted feature of the information part. Step S025S: Multiply the non-information mask and the gate weights pixel by pixel and then concatenate them with the input feature map to obtain the weighted features of the non-information part; Step S026S: After splitting the weighted features of the information part in half according to the channel dimension, we obtain the features of the first information part and the features of the second information part. After splitting the weighted features of the non-information part in half according to the channel dimension, we obtain the features of the first non-information part and the features of the second non-information part. Step S027S: The output of the pixel-by-pixel summation of the features of the first information part and the features of the second non-information part is concatenated with the output of the pixel-by-pixel summation of the features of the second information part and the features of the first non-information part according to the channel dimension to obtain the output of the spatial dimension reconstruction convolution module.

5. The method for transparent object recognition based on an RGB-D feature fusion enhancement network according to claim 3, characterized in that, The content of the channel dimension reconstruction convolution module is as follows: Step S021C: Take the output of the spatial dimension reconstruction convolutional module as input and split it into two parts, up feature map and low feature map, according to the number of channels of the output of the spatial dimension reconstruction convolutional module; Step S022C: After performing channel compression on the up feature map, the up feature map is divided into two parts. One part is processed by a 3×3 depthwise DW convolution, and the other part is processed by a 1×1 pointwise PW convolution. Then, pixel-wise addition is performed to obtain the up_refined feature map. Step S023C: After performing channel compression on the low feature map, the output after a 1×1 pointwise PW convolution is concatenated with the low feature map along the channel dimension to obtain the low_fused feature map. Step S024C: Concatenate the up_refined feature map and the low_fused feature map along the channel dimension to obtain the concatenated feature map. Multiply the output after adaptive average pooling and softmax activation function with the concatenated feature map pixel by pixel to obtain the output feature map. Step S025C: Split the output feature map into two parts according to the number of channels, and add the two parts pixel by pixel to obtain the output of the final channel dimension reconstruction convolution module.

6. The method for transparent object recognition based on an RGB-D feature fusion enhancement network according to claim 1, characterized in that, The specific content of FAM mentioned in step S02 is as follows: Step S021F: After passing through the channel dimension enhancement module, the shallow feature map is multiplied pixel by pixel with the shallow feature map to obtain the channel dimension enhancement coefficient map of the shallow feature map. Step S022F: After the deep feature map passes through the channel dimension enhancement module, it is multiplied pixel by pixel with the deep feature map to obtain the channel dimension enhancement coefficient map of the deep feature map; Step S023F: After passing the channel dimension enhancement coefficient map of the shallow feature map through the spatial dimension enhancement module, multiply it pixel by pixel with the channel dimension enhancement coefficient map of the shallow feature map to obtain the spatial dimension enhancement coefficient map of the shallow feature map. Step S024F: After passing the channel dimension enhancement coefficient map of the deep feature map through the spatial dimension enhancement module, multiply it pixel by pixel with the channel dimension enhancement coefficient map of the deep feature map to obtain the spatial dimension enhancement coefficient map of the deep feature map. Step S025F: Finally, the spatial dimension enhancement coefficient map of the shallow feature map and the spatial dimension enhancement coefficient map of the deep feature map are concatenated along the channel dimension to obtain the final output of the model.

7. The method for transparent object recognition based on an RGB-D feature fusion enhancement network according to claim 6, characterized in that, The channel dimension enhancement module consists of the following steps: dividing the input feature map into two parts; one part undergoes adaptive average pooling, channel compression, and dimension order adjustment to obtain an average feature map; the other part undergoes adaptive max pooling, channel compression, and dimension order adjustment to obtain a global average feature map; the average feature map and the global average feature map are concatenated along the channel dimension, and then subjected to a 5×5 convolution and a sigmoid activation function to obtain a channel dimension enhancement coefficient map.

8. The method for transparent object recognition based on an RGB-D feature fusion enhancement network according to claim 6, characterized in that, The spatial dimension enhancement module is as follows: taking the output feature map of the channel dimension enhancement module as input, the input feature map is divided into two parts, one part is subjected to average pooling and the other part is subjected to max pooling. After concatenating the output of average pooling and the output of max pooling in the channel dimension, the result is a channel dimension enhancement coefficient map after passing through a 5×5 convolution and a sigmoid activation function.

9. A device for transparent object recognition based on an RGB-D feature fusion enhancement network, characterized in that, The device implements the method as described in any one of claims 1 to 8, comprising: Image acquisition module: used to acquire RGB-D images of transparent objects, and preprocess the depth image D2 obtained by RGB and depth estimation of the image to obtain an RGB-D2 bimodal aligned image; Model building module: used to build an improved YOLOV8-seg network: design the C2f_ScConv module to replace the C2f module in the backbone of the original network; design the global hybrid feature attention enhancement module HAM and insert it after the SPPF module in the backbone of the original network; design the multi-dimensional feature fusion module FAM to replace the Concat module in the neck of the original network. The specific content of the HAM is as follows: Step S021H: After passing the depth image D2 through a 3×3 convolution, a BN layer, a GELU activation function, a max pooling layer, a 3×3 convolution, a BN layer, and a GELU activation function in sequence, the output is bilinearly interpolated and aligned with the RGB features in the resolution dimension to obtain the depth prior features; Step S022H: The RGB features are passed through a 1×1 convolution to align with the depth prior features in the channel dimension. After being concatenated with the depth prior features in the channel dimension, they are passed through a 3×3 convolution, a BN layer, and a GELU activation function to obtain the shallow fusion features. Step S023H: Adjust the dimensional order of the shallow fusion features: [batch size b, channel c, height H, width W] to: [batch size b, channel c, h*window_size, w*window_size], and then rearrange them to: [batch size b*h*w, window_size*window_size, channel c]; Step S024H: The rearranged shallow fusion features are processed through a multi-head self-attention mechanism to obtain a self-attention map, and the self-attention map is rearranged from the dimensions of [batch size b*h*w, window_size* window_size, channel c] to: [batch size b, channel c, height H, width W]. Step S025H: Multiply the rearranged self-attention map with the RGB features pixel by pixel, and then add it to the RGB features pixel by pixel to obtain the final output.

Images