A transparent object detection method, apparatus and device

Abstract

The application discloses a transparent object detection method, device and equipment, and relates to the technical field of transparent object detection. The method comprises the following steps: extracting semantic features through a residual network module; extracting boundary features through a boundary detection module; fusing the semantic features and the boundary features through a fusion layer to obtain semantic-boundary fusion features; mapping the semantic features and the boundary features into semantic queries and boundary queries respectively through a cross-attention mechanism layer, mapping the semantic-boundary fusion features into semantic keys and semantic values corresponding to the semantic queries, boundary keys and boundary values corresponding to the boundary queries, and weighting the semantic features and the boundary features through the cross-attention mechanism based on the obtained queries, keys and values to obtain weighted semantic features and weighted boundary features; and decoding the weighted semantic features and the weighted boundary features through a decoding module to obtain transparent objects in a to-be-detected image.

Description

Technical Field

[0001] This invention relates to the field of transparent object detection technology, and in particular to a method, apparatus and equipment for detecting transparent objects. Background Technology

[0002] Transparent object detection is a challenging problem in computer vision. Service robots need to accurately identify transparent objects, such as glasses, transparent bags, or plastic boxes, for safe grasping and handling. Due to their physical properties, the reflection and refraction of light by transparent objects result in unclear boundaries and a lack of sufficient texture information in images, making traditional image processing methods difficult to effectively identify and locate them. With the development of deep learning technology, deep learning-based methods have been widely applied to transparent object detection due to their high efficiency in image recognition and processing.

[0003] Current methods for detecting transparent objects primarily use convolutional neural networks (CNNs) to automatically learn the features of transparent objects in images. However, this approach limits the receptive field of the network model, making it difficult to recognize and detect transparent objects in complex scenes. The paper "Segmenting transparent object in the wild with transformer" proposes the visual segmentation method Trans2Seg, which is the first to apply the Transformer to transparent object segmentation. This method first feeds the image into a CNN-based encoder to extract features, then feeds these features into a Transformer module for self-attention operations. The enhanced features are then fed into a decoder to obtain the final segmentation result. However, this method does not fully utilize the boundary information of transparent objects, resulting in uneven edges and a low feature matching rate. The paper "A Visual Detection Method for Autonomous Vehicles Considering the Influence of Transparent Objects" also addresses this issue. It first extracts features from the input image, then passes it through a boundary detection module to extract the edge information of transparent objects. The edge features are then fused and corrected with the results of the initial segmentation module, and finally, a Transformer-based encoder-decoder mechanism is used to obtain the prediction result.

[0004] However, the above method only fuses and corrects the semantic information and boundary information of transparent objects in the last step, ignoring the potential mutual reinforcement relationship between the two. This can easily lead to the model's misunderstanding of the boundaries of transparent objects, resulting in the need to improve the detection accuracy of transparent objects. Summary of the Invention

[0005] Based on this, in order to solve the technical problems in the prior art, the present invention provides a method, apparatus and device for detecting transparent objects.

[0006] This invention provides a method for detecting transparent objects, comprising:

[0007] A neural network is constructed, comprising a parallel residual network module and a boundary detection module, a semantic-boundary co-evolution module connected to the outputs of the residual network module and the boundary detection module, and a decoding module connected to the output of the semantic-boundary co-evolution module; wherein the semantic-boundary co-evolution module comprises a fusion layer and a cross-attention mechanism layer connected in sequence.

[0008] Images containing transparent objects are collected to build a dataset. The dataset is then used to train a neural network to obtain a transparent object detection model.

[0009] The detection model inputs an image containing transparent objects into the detection model. A residual network module extracts semantic features of the transparent objects in the image. An edge detection algorithm in the boundary detection module extracts boundary features of the transparent objects. A fusion layer in the semantic-boundary co-evolution module fuses the semantic and boundary features to obtain semantic-boundary fusion features of the transparent objects. A cross-attention mechanism layer in the semantic-boundary co-evolution module maps the semantic and boundary features to semantic queries and boundary queries, respectively. The semantic-boundary fusion features are mapped to semantic keys and values ​​corresponding to semantic queries, and boundary keys and values ​​corresponding to boundary queries. Based on the obtained queries, keys, and values, a cross-attention mechanism is applied to weight the semantic and boundary features to obtain weighted semantic and boundary features. Finally, a decoding module decodes the weighted semantic and boundary features to obtain the transparent objects in the image.

[0010] Furthermore, the step of mapping semantic features and boundary features to semantic queries and boundary queries respectively, and mapping semantic-boundary fusion features to semantic keys and semantic values ​​corresponding to semantic queries and boundary keys and boundary values ​​corresponding to boundary queries, specifically includes:

[0011] The semantic features, boundary features, and semantic-boundary fusion features that are uniformly transformed into C×H×W are flattened into 2D semantic features F. s '、2D boundary features F e 'and 2D semantic-boundary fusion feature F c Among them, F s '、F e 'and F c '∈R N×C N = H × W, where C is the number of channels;

[0012] The 2D semantic features F are respectively processed by the pre-trained semantic mapping matrix. s 'and 2D semantic-boundary fusion feature F c Perform mapping to obtain semantic query Q sSemantic key K s and semantic value V s :

[0013]

[0014] in, and The semantic mapping matrix is ​​pre-trained.

[0015] The 2D boundary features F are respectively processed by the pre-trained boundary mapping matrix. e 'and 2D semantic-boundary fusion feature F c Perform mapping to obtain boundary query Q e Boundary key K e and boundary value V e :

[0016]

[0017] in, and This is a pre-trained boundary mapping matrix.

[0018] Furthermore, the boundary features are weighted using a cross-attention mechanism, specifically including:

[0019] Boundary query Q e Boundary key K e and boundary value V e The dimensions are reshaped to H×W×C, and the reshaped boundary query f is obtained. Q Reshape boundary key f K and reshaping boundary value f V ;

[0020] In the H×W dimension, the boundary query f will be reshaped respectively. Q and reshaping boundary key f K The system is divided into n×n regions, and average pooling is performed on each region to obtain a first weight representation matrix Q and a second weight representation matrix K. Based on the first weight representation matrix Q and the second weight representation matrix K, the region matching score matrix G is obtained.

[0021] G = Q(K) T

[0022] The superscript T indicates the transpose operation;

[0023] In each row of the region matching score matrix G, select the top k scores in the score ranking, integrate the indices of the k scores to obtain matrix S; and retrieve the reshaping boundary key f based on the index. K and reshaping boundary value f V The corresponding area in the middle:

[0024] f′ K =Gather(f K ,S)

[0025] f′ V =Gather(f V ,S)

[0026] Here, `Gather(·)` is the `torch.gather` operation, used to extract elements from the input tensor along a specified dimension based on the index and construct a new tensor; f' K and f' V They are respectively with f Q The key features and value features with the highest correlation;

[0027] f' K and f' V Calculate spatial attention A after transforming into a 2D shape e :

[0028]

[0029] Among them, K' e and V′ e These are f' K and f' V The 2D flattened bond features and 2D flattened value features obtained after transforming into a 2D shape; d = C / h, where h is the number of attention heads;

[0030] The 2D boundary feature F is connected via residual connection. e Adding Spatial Attention A e The calculation results yield the weighted boundary feature f. ei :

[0031]

[0032] in, It is the element-wise addition operation in residual join.

[0033] Furthermore, the semantic features are weighted using a cross-attention mechanism, specifically including:

[0034] Using cross-attention mechanism to V s We perform weighted calculations to obtain the attention score A. s :

[0035]

[0036] Where d = C / h, h is the number of attention heads; the superscript T indicates the transpose operation;

[0037] The 2D semantic features F are connected via residual connections.s 'Add attention score A' s The calculation results yield the weighted semantic features f. si :

[0038]

[0039] in, It is the element-wise addition operation in residual join.

[0040] Furthermore, the output of the cross-attention mechanism layer in the semantic-boundary co-evolution module is connected to a Mix-FFN network layer; the weighted semantic features and weighted boundary features output by the cross-attention mechanism layer are enhanced through the Mix-FFN network layer, specifically including:

[0041]

[0042] Where MPL() is the feedforward operation in the Mix-FFN network, Conv 3×3 () represents the convolution operation in the Mix-FFN network; GeLU() represents the activation function operation in the Mix-FFN network; f ei It is a weighted boundary feature, f eo It enhances boundary features; f si It is a weighted semantic feature, f so It enhances semantic features.

[0043] Furthermore, the residual network module is formed by a ResNeXt-101 network; multiple residual convolutional layers of the ResNeXt-101 network are used to extract multi-scale features F of transparent objects in the image to be detected. i , i∈{0,1,2,3,4,5}, the feature F5 with the most semantic information is used as the semantic feature of transparent objects in the image to be detected.

[0044] Furthermore, a boundary refinement module is connected to the output of the boundary detection module and the output of the residual network module. The boundary refinement module includes an aggregation layer, a residual layer, a spatial attention mechanism layer, and a fusion layer. The boundary refinement module refines the boundary features output by the boundary detection module, specifically including…

[0045] The features F0, F1, F2, and F3 output by the ResNeXt-101 network are upsampled to the same dimension and then aggregated with the boundary features through an aggregation layer.

[0046] f i =Upsample(F i ), i = 0, 1, 2, 3

[0047] sum=[f0,f1,f2,f3,f m ]

[0048] Where Upsample() is the upsampling operation, f m These are the boundary features extracted by the Canny edge detection algorithm in the boundary detection module; [] represents the feature aggregation operation;

[0049] The upsampled feature map f0, which has the most boundary features but the least semantic information, is enhanced by the residual layer:

[0050]

[0051] Where ReLU() is the activation function, BN() is the batch normalization operation; Conv 3×3 () is the convolution operation, and f'0 is the enhanced boundary feature; It is the element-wise addition operation in the residual connection; f'0 is the initial reinforcement boundary feature f'0;

[0052] The initial enhanced boundary feature f'0 is further enhanced through a spatial attention mechanism layer:

[0053]

[0054] α j =Sigmoid([f'0, f)) j j∈{1, 2, sum}

[0055] Where Sigmoid() is the sigmoid function used for normalization, α j The attention map is obtained from the j-th upsampled feature map. This is an element-wise multiplication operation; This is the enhanced j-th boundary feature;

[0056] The fourth boundary feature output by the spatial attention mechanism layer is incorporated through a fusion layer. Feature aggregation is performed on F5, the feature with the most semantic information output by the ResNeXt-101 network, to obtain refined boundary features.

[0057]

[0058] Furthermore, the semantic-boundary co-evolution module includes a first semantic-boundary co-evolution module, a second semantic-boundary co-evolution module, a third semantic-boundary co-evolution module, and a fourth semantic-boundary co-evolution module connected in sequence; the first, second, third, and fourth semantic-boundary co-evolution modules enhance the semantic features and boundary features in the feature transfer process by being connected in series.

[0059] This invention provides a transparent object detection device, comprising:

[0060] A model building module is used to build a neural network, which includes a parallel residual network module and a boundary detection module, a semantic-boundary co-evolution module connected to the output of the residual network module and the boundary detection module, and a decoding module connected to the output of the semantic-boundary co-evolution module; wherein, the semantic-boundary co-evolution module includes a fusion layer and a cross-attention mechanism layer connected in sequence.

[0061] The model training module is used to collect images containing transparent objects to build a dataset, and to train the neural network using the dataset to obtain a transparent object detection model.

[0062] The detection module takes an image containing transparent objects as input into the detection model. It extracts semantic features of the transparent objects using a residual network module; extracts boundary features of the transparent objects using an edge detection algorithm in the boundary detection module; fuses semantic and boundary features using a fusion layer in the semantic-boundary co-evolution module to obtain semantic-boundary fusion features of the transparent objects; maps semantic and boundary features to semantic queries and boundary queries using a cross-attention mechanism layer in the semantic-boundary co-evolution module, mapping the semantic and boundary features to semantic keys and values ​​corresponding to semantic queries and boundary keys and values ​​corresponding to boundary queries. Based on the obtained queries, keys, and values, the semantic and boundary features are weighted using a cross-attention mechanism to obtain weighted semantic and boundary features; and decodes the weighted semantic and boundary features using a decoding module to obtain the transparent objects in the image to be detected.

[0063] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described transparent object detection method.

[0064] The above-mentioned at least one technical solution adopted in this invention can achieve the following beneficial effects:

[0065] This invention fuses semantic features and boundary features. By matching queries based on semantic features with key values ​​obtained from the fused features, the semantic features acquire contextual information related to the boundary features from the fused features, thereby capturing detailed semantic information about the boundaries of transparent objects. Similarly, by matching queries based on boundary features with key values ​​obtained from the fused features, the boundary information gains more contextual support from the semantic features, making the pixels around the boundary features more continuous. The attention mechanism uses query-key-value attention weighting, allowing semantic and boundary information to mutually enhance each other, promoting the co-evolution of semantic and boundary features. That is, semantic features can enhance their semantic expression through boundary information, while boundary features can also use semantic information to more accurately locate and describe the boundary. This mutual interaction enhances the model's understanding of the boundaries of transparent objects, resulting in more comprehensive and accurate transparent object features and thus more accurate detection results. Attached Figure Description

[0066] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0067] Figure 1 This is a schematic diagram of the transparent object detection method provided by the present invention;

[0068] Figure 2 A schematic diagram of the neural network framework provided for this invention;

[0069] Figure 3 A schematic diagram of the boundary refinement module provided by the present invention;

[0070] Figure 4 A schematic diagram of the semantic-boundary co-evolution module based on the cross-attention mechanism provided by the present invention;

[0071] Figure 5 This is a schematic diagram of the detection results of transparent objects provided by the present invention;

[0072] Figure 6 This is a schematic diagram of the detection results of transparent objects provided by the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this invention.

[0074] Existing methods for detecting transparent objects do not consider that semantic and boundary information in image features can mutually enhance each other. Using boundary features to strengthen semantic features helps semantic features acquire more detailed information, while using semantic features to strengthen boundary features makes pixels around the boundary features more continuous, thus improving the model's ability to distinguish foreground from background. Therefore, this invention proposes to co-evolve boundary features and semantic features, resulting in smoother, more complete, and more robust segmentation results.

[0075] To address the shortcomings of existing methods, this invention proposes a transparent object detection method based on the co-evolution of semantic and boundary features, focusing on solving the problem of transparent objects blending with the background in complex scenes, making complete segmentation difficult. This invention uses a Transformer to expand the model's receptive field and employs a cross-attention mechanism to promote the co-evolution of semantic and boundary features. First, a convolutional neural network is used to extract high-dimensional features from the image. Shallow features are fused to form a boundary feature stream, and deep features are fused to form a semantic feature stream. Then, a semantic-boundary co-evolution module is used to mutually enhance semantic and boundary features. To reduce noise in the boundary feature stream, a boundary refinement module based on multi-scale feature fusion is introduced to refine the boundary features. Finally, the results are input into a decoder to generate the segmentation result. This invention proposes the co-evolution of boundary and semantic features, resulting in a smoother, more complete, and more robust segmentation result.

[0076] Example 1

[0077] Figure 1 The flowchart of the transparent object detection method of this embodiment is shown. The following details the process. Figure 1 This method is explained in detail, specifically including the following steps:

[0078] S1: Construct a neural network, which includes a parallel residual network module and a boundary detection module, a semantic-boundary co-evolution module connected to the outputs of the residual network module and the boundary detection module, and a decoding module connected to the output of the semantic-boundary co-evolution module; wherein, the semantic-boundary co-evolution module includes a fusion layer and a cross-attention mechanism layer connected in sequence. The input of the first semantic-boundary co-evolution module is connected to the outputs of the residual network module and the boundary refinement module, using the features output by the ResNeXt-101 network and the refined boundary features output by the boundary refinement module as input; the output of the fourth semantic-boundary co-evolution module is used as the input of the decoding module.

[0079] The constructed neural network structure is as follows Figure 2 As shown, the network includes a residual network module formed by the ResNeXt-101 network, a boundary detection module formed by the Canny edge detection algorithm, a boundary refinement module formed by an aggregation layer, a residual layer, a spatial attention mechanism layer, and a fusion layer, four semantic-boundary co-evolutionary modules connected in series, and a decoding module for detecting transparent object boundaries. The input of the boundary refinement module is connected to the outputs of the residual network module and the boundary detection module, and the input of the first semantic-boundary co-evolutionary module is connected to the outputs of the residual network module and the boundary refinement module. This neural network employs a custom feature fusion and evolution-based network framework. It learns features from a large sample set through pre-training on the ImageNet dataset, enabling multi-scale image feature extraction and more accurate extraction of high-level image features even in more generalized scenarios. The working flow of this neural network is as follows:

[0080] (1) The input image of size 3×H×W is passed through a replaceable backbone feature extraction network to obtain a feature map of size C×H×W containing high-level information, where C is the number of channels in the feature map. The replaceable backbone feature extraction network can be any deep neural network-based backbone feature extraction network. Deep neural networks have made great progress in image information extraction, and this invention uses the ResNeXt-101 network. Multi-scale features F are extracted from the shallow features to the deep features of this network. i , i∈{0,1,2,3,4,5}, are used for multi-scale feature fusion. ResNeXt-101 is an improved neural network based on the Residual Network (ResNet) architecture, which performs well in image classification and other computer vision tasks. ResNeXt introduces the concept of "grouped convolution," improving performance by increasing the width rather than the depth of the original ResNet network. The advantage of this structure is that it reduces the increase in the number of parameters while improving the network's expressive power and computational efficiency.

[0081] (2) The multi-layer features extracted from the backbone feature extraction network and the image boundary f m The data is fed into the boundary refinement module to obtain the initial features F1 of the boundary feature flow. e .

[0082] (3) F1 e f5 is used as the input feature of the boundary branch and semantic branch of the first semantic-boundary co-evolutionary module, respectively. These are refined to obtain the refined boundary feature stream and semantic feature stream, which are then used as the feature input of the next semantic-boundary co-evolutionary module. After feature refinement by four semantic-boundary co-evolutionary modules, the final semantic feature stream is obtained. This stream is then passed through a CNN-based decoder to obtain the final result.

[0083] The structure of each module in the neural network is explained in detail below.

[0084] Residual network module.

[0085] Formed by a ResNeXt-101 network; multi-scale features F of transparent objects in the image to be detected are extracted through multiple residual convolutional layers of the ResNeXt-101 network. i , i∈{0,1,2,3,4,5}, the feature F5 with the most semantic information is used as the semantic feature of transparent objects in the image to be detected.

[0086] Boundary refinement module.

[0087] like Figure 3 As shown, this invention employs a custom-designed boundary refinement module based on multi-scale feature fusion. It uses multi-scale features extracted from the backbone feature extraction network for aggregation and progressive refinement operations to generate boundary refinement features for transparent objects while reducing background noise. Considering that deep features possess rich high-level semantic features and shallow features possess rich fine-grained features, aggregating features from different levels can fully utilize contextual information. The specific implementation is as follows:

[0088] The aggregation layer upsamples the multi-scale features output by the ResNeXt-101 network to the same dimension, facilitating the aggregation operation.

[0089] f i =Upsample(F i ), i = 0, 1, 2, 3.

[0090] Upsample is the upsampling operation. Then, multi-scale features are aggregated:

[0091] f sum =[f0, f1, f2, f3, f m ]

[0092] Where Upsample() is the upsampling operation, f m These are the boundary features extracted by the Canny edge detection algorithm in the boundary detection module; [] represents the feature aggregation operation.

[0093] By utilizing spatial attention, deep features are continuously refined into shallow features, removing noise from the shallow features and further refining the pixels around the boundaries of transparent objects. The specific steps are as follows:

[0094] Enhance shallow features f0 using residual convolution modules:

[0095]

[0096] Where ReLU() is the activation function, BN() is the batch normalization operation; Conv 3×3 () is the convolution operation, and f'0 is the enhanced boundary feature; This is the element-wise addition operation in the residual connection; f'0 is the initial enhanced boundary feature f'0. The residual module can further enhance the feature representation without discarding the original feature details.

[0097] Use f1, f2, f in sequence sum Enhancement is achieved by using spatial attention operations to strengthen boundary features. First, an attention map α is generated:

[0098] α j =Simgoid([f'0, f j j∈{1, 2, sum};

[0099] Where Sigmoid() is the sigmoid function used for normalization, α j This is the attention map obtained from the j-th upsampled feature map. Then, attention map α is used to enhance f'0:

[0100]

[0101] in, For element-wise multiplication, This is the enhanced j-th boundary feature. f1, f2, f... are used sequentially. sum After repeating the above operations, we get

[0102] The fourth boundary feature output by the spatial attention mechanism layer is incorporated through a fusion layer. Feature aggregation is performed on F5, the feature with the most semantic information output by the ResNeXt-101 network, to obtain the refined boundary features.

[0103]

[0104] This feature will be used as the initial feature of the boundary feature stream and will be sent to the first semantic-boundary co-evolution module for enhancement.

[0105] Semantic-boundary co-evolution module.

[0106] The structure of the semantic-boundary co-evolutionary module is as follows: Figure 4 As shown. The initial features F1 of the boundary feature flow are... e (i.e., refined boundary features) and initial features F1 of semantic feature flow s (i.e., F5) serves as the input to the first semantic-boundary co-evolutionary module. For each semantic-boundary co-evolutionary module, the boundary feature flow F is first processed. e With semantic feature flow F s Feature aggregation is performed to obtain F c :

[0107] F c =[F s ,F e ].

[0108] F e F s F c After the shape is uniformly transformed into C×H×W, it is serialized into a flattened 2D feature F. s ',F e ',F c '∈Ρ N×C N = H × W, where C is the number of channels.

[0109] The semantic-boundary co-evolutionary module is divided into a boundary branch and a semantic branch. The input of the boundary branch is the boundary feature flow F. e The input to the semantic branch is the semantic feature flow F. s Using a mapping matrix to map 2D features, for boundary branches:

[0110] Q e =F e 'W e Q ,K e =F c 'W e K V e =F c 'W e V .

[0111] and This is a pre-trained semantic mapping matrix. Similarly, for semantic branches:

[0112] Qs =F s 'W s Q ,K s =F c 'W s K V s =F c 'W s V .

[0113] in, and This is a pre-trained boundary mapping matrix.

[0114] For semantic branches, use a cross-attention mechanism:

[0115]

[0116] d = C / h, where h is the number of attention heads. Using residual connections, add F... s ':

[0117]

[0118] The feature location information is provided using Mix-FFN from "SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers", as follows:

[0119]

[0120] Get f so Then, the enhanced semantic features are restored to their original shape and used as input to the semantic feature stream of the next semantic-boundary co-evolution module. Mix-FFN no longer uses positional encoding in ViT to provide positional information, as positional encoding is not necessary for semantic segmentation. Instead, it uses 3×3 convolutions in the feedforward network FFN to provide positional information.

[0121] For boundary branches, a custom region dynamic selection mechanism is used.

[0122] Boundary query Q e Boundary key K e and boundary value V e The dimensions are reshaped to H×W×C, denoted as f. Q f K f V Then, it is divided into n×n regions along the H×W dimension, and Q is... e and K eAfter the partitioning operation, each region undergoes average pooling to obtain a first weight representation matrix Q and a second weight representation matrix K. Finally, the region matching score matrix is ​​calculated.

[0123] G = Q(K) T .

[0124] In each row of G, select the k highest scores and record their indices to obtain matrix S. Then, retrieve V based on the indices. e With K e The corresponding area in the middle:

[0125] f′ K =Gather(f K ,S),f′ V =Gather(f V ,S).

[0126] Where Gather(·) is the torch.gather operation. The resulting f′ K and f' V Transforming it into a 2D shape is denoted as K' e ,V' e This facilitates subsequent attention-based operations.

[0127] The spatial attention is calculated as follows:

[0128]

[0129] Next, the original boundary feature flow F e 'join in:

[0130]

[0131] Similar to semantic branching, the final boundary branch output is obtained:

[0132]

[0133] f eo This refers to the enhanced boundary features.

[0134] In summary, the entire neural network workflow is as follows: An input image of size 3×H×W is fed into the ResNeXt-101 network to extract multi-scale features F0, F1, F2, F3, F4, and F5. The semantically rich F5 is used as the initial feature F1 for the semantic feature stream. s Four cascaded semantic boundary co-evolutionary modules (SBMA) are used to enhance the two feature streams. The input to each SBMA module is the semantic feature stream F. i s and boundary feature flow F i e, where i = 1, 2, 3, 4. The output of the previous SBMA becomes the input of the next SBMA.

[0135] For the boundary characteristic flow F i e The expression is as follows:

[0136]

[0137] MSBR refers to the boundary refinement module, f m It is the image boundary extracted using a boundary detection algorithm. Let [i] be the boundary branch of the i-th SBMA, and [] be the feature aggregation operation, as shown in the following equation:

[0138] [a,b]=Conv 3×3 (Concat(a,b));

[0139] Conv 3×3 () refers to a convolution operation using a 3×3 kernel, while Concat() refers to a concatenation operation based on the channel dimension.

[0140] For semantic feature flow F i s The expression is as follows:

[0141]

[0142] F i sum The sum of boundary flow features and semantic flow features is:

[0143]

[0144] in It involves element-wise addition. After the fourth SBMA, the semantic feature stream is decoded to generate the final result. Addressing the limitations of existing methods, such as limited receptive field and incomplete segmentation of transparent objects in complex scenes, this invention proposes a boundary refinement module based on multi-scale feature fusion to effectively extract the blurred boundaries of transparent objects. This module utilizes the characteristics of features at different levels to refine the boundary information of transparent objects while removing background noise. To enhance the completeness of transparent object segmentation in complex scenes, a semantic-boundary co-evolution module based on a cross-attention mechanism is designed. This module facilitates the interaction between semantic information and the image, and works simultaneously with a designed dynamic region selection mechanism to achieve the co-evolution of semantic and boundary information.

[0145] S2: Collect images containing transparent objects to build a dataset, use the dataset to train the neural network, and obtain a transparent object detection model.

[0146] The custom multi-scale feature loss function and the final loss function specifically include:

[0147] (1) Calculate the cross-entropy loss L between the output of the semantic branch of each semantic-boundary co-evolutionary module and the real label. s .

[0148] (2) Calculate the cross-entropy loss L between the output of the boundary branch of each semantic-boundary co-evolutionary module and the true boundary label. b The true boundary label is obtained by selecting 8 pixel values ​​from the boundary of the true label as the true boundary value.

[0149] (3) Calculate the cross-entropy loss between the final result and the true label. The final loss function is as follows:

[0150]

[0151] Where λ s =0.01, λ b =0.25.

[0152] Experimental results show that this method can improve the segmentation effect of transparent objects to a certain extent, and its performance is superior to state-of-the-art methods. The experimental results analysis is given below:

[0153] The experimental results were analyzed using indices mIoU and F, which are specific to the field of transparent object detection. β The evaluation was based on mAE and mBER. Comparative experiments were conducted on three datasets: GSD, HSO, and GDD.

[0154] The comparison methods included MirrorNet, GDNet, TransLab, Trans2Seg, EBLNet, GSDNet, PGSNet, and RFENet. The test results on the GSD, GDD, and HSO datasets are shown in Tables 1, 2, and 3, respectively.

[0155] Table 1 shows the test results on the GSD dataset.

[0156]

[0157]

[0158] Table 2 Test results on the GDD dataset

[0159] Method mIoU <![CDATA[F β ]]> mAE mBER MirrorNet 85.07 0.866 0.083 7.67 GDNet 87.63 0.898 0.063 5.62 TransLab 81.64 0.849 0.097 9.70 Trans2Seg 84.41 0.872 0.078 7.36 GSDNet 88.07 0.932 0.059 5.71 PGSNet 87.81 0.901 0.062 5.56 GDNet-B 87.83 0.939 0.061 5.52 RFENet 87.84 0.940 0.063 6.37 GlassFormer (Ours) 88.60 0.943 0.056 5.51

[0160] Table 3 shows the test results on the HSO dataset.

[0161]

[0162]

[0163] Experimental results show that this invention outperforms existing methods on a large number of benchmark datasets for transparent object detection. Some visualizations of the segmentation results are shown below. Figure 5 and Figure 6 , Figure 5 The first column is the original image including transparent objects, and the second to seventh columns are the detection results obtained using the Trang2Seg model, GDNet model, EBLNet model, GSDNet model, RFENet model, the model of this invention, and GT model, respectively. Figure 6 The first column shows the original image including the transparent object. The second to fourth columns show the detection results obtained using the TransLab model, EBLNet model, the model described in this invention, and the GT model, respectively. It can be observed that in many complex scenes, such as when the background and the boundary of the transparent object are blended together, or when similar transparent objects exist in the background, the segmentation effect and robustness of this invention are significantly improved. This is because this invention has a global receptive field and adaptively selects local and global features to mutually enhance boundary and semantic features. This invention fully utilizes the co-evolution of local fine-grained features and high-level semantic features, enabling the network to uncover deeper differences between transparent and background objects without losing local details. Therefore, it can better distinguish between foreground and background, resulting in stronger robustness.

[0164] S3: Input the image to be detected containing transparent objects into the detection model. Extract the semantic features of the transparent objects in the image through the residual network module. Extract the boundary features of the transparent objects in the image through the edge detection algorithm in the boundary detection module. Fuse the semantic features and boundary features through the fusion layer in the semantic-boundary co-evolution module to obtain the semantic-boundary fusion features of the transparent objects. Map the semantic features and boundary features to semantic queries and boundary queries respectively through the cross-attention mechanism layer in the semantic-boundary co-evolution module. Map the semantic features and boundary features to semantic keys and semantic values ​​corresponding to semantic queries and boundary keys and boundary values ​​corresponding to boundary queries. Based on the obtained queries, keys, and values, apply the cross-attention mechanism to weight the semantic features and boundary features respectively to obtain weighted semantic features and weighted boundary features. Decode the weighted semantic features and weighted boundary features through the decoding module to obtain the transparent objects in the image to be detected.

[0165] This invention discloses a transparent object detection method based on the co-evolution of semantic and boundary features, further promoting effective context and boundary learning in transparent object images. It includes: proposing a novel Transformer-based learning framework based on the co-evolution of semantic and boundary features, which adopts a boundary-semantic dual-stream structure, utilizing the complementary and mutually reinforcing features of both. To achieve this, a semantic-boundary co-evolution module based on a cross-attention mechanism is proposed, which utilizes the cross-attention mechanism to achieve interaction and fusion of boundary and semantic features. To effectively utilize local and global features, a dynamic region selection mechanism is proposed, causing the boundary feature stream to focus more on the boundary of the transparent object, mining more detailed features, and effectively improving the integrity and smoothness of the transparent object boundary. To obtain purer boundary features, a boundary refinement module based on multi-scale feature fusion is proposed, utilizing multi-scale features extracted from the backbone network and image boundary information, and continuously refining the boundary information using feature aggregation and feature refinement strategies. Experiments demonstrate that this method can effectively and perfectly detect transparent objects in complex scenes, exhibiting strong robustness.

[0166] The above describes a transparent object detection method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding transparent object detection device, including:

[0167] The model building module is used to build a neural network, which includes a parallel residual network module and a boundary detection module, a semantic-boundary co-evolution module connected to the output of the residual network module and the output of the boundary detection module, and a decoding module connected to the output of the semantic-boundary co-evolution module; wherein, the semantic-boundary co-evolution module includes a fusion layer and a cross-attention mechanism layer connected in sequence.

[0168] The model training module is used to collect images containing transparent objects to build a dataset, and then use the dataset to train the neural network to obtain a transparent object detection model.

[0169] The detection module takes an image containing transparent objects as input into the detection model. It extracts semantic features of the transparent objects using a residual network module; extracts boundary features of the transparent objects using an edge detection algorithm in the boundary detection module; fuses semantic and boundary features using a fusion layer in the semantic-boundary co-evolution module to obtain semantic-boundary fusion features of the transparent objects; maps semantic and boundary features to semantic queries and boundary queries using a cross-attention mechanism layer in the semantic-boundary co-evolution module, mapping the semantic and boundary features to semantic keys and values ​​corresponding to semantic queries and boundary keys and values ​​corresponding to boundary queries. Based on the obtained queries, keys, and values, the semantic and boundary features are weighted using a cross-attention mechanism to obtain weighted semantic and boundary features; and decodes the weighted semantic and boundary features using a decoding module to obtain the transparent objects in the image to be detected.

[0170] Specific limitations regarding the transparent object detection device can be found in the limitations of the transparent object detection method described above, and will not be repeated here. Each module in the aforementioned transparent object detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0171] This invention also provides the structure of a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for various operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above-mentioned functions. Figure 1 A method for detecting transparent objects is provided.

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0173] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

Claims

1. A method for detecting transparent objects, characterized in that, include: A neural network is constructed, comprising a parallel residual network module and a boundary detection module, a semantic-boundary co-evolution module connected to the outputs of the residual network module and the boundary detection module, and a decoding module connected to the output of the semantic-boundary co-evolution module; wherein the semantic-boundary co-evolution module comprises a fusion layer and a cross-attention mechanism layer connected in sequence. Images containing transparent objects are collected to build a dataset. The dataset is then used to train a neural network to obtain a transparent object detection model. The detection model inputs an image containing transparent objects into the detection model. Semantic features of the transparent objects in the image are extracted using a residual network module. Boundary features of the transparent objects in the image are extracted using an edge detection algorithm in the boundary detection module. The semantic and boundary features are fused using a fusion layer in the semantic-boundary co-evolution module to obtain semantic-boundary fusion features of the transparent objects. The semantic and boundary features are mapped to semantic queries and boundary queries respectively using a cross-attention mechanism layer in the semantic-boundary co-evolution module. The semantic-boundary fusion features are mapped to semantic keys and semantic values ​​corresponding to semantic queries, and boundary keys and boundary values ​​corresponding to boundary queries. Based on the obtained queries, keys, and values, the semantic and boundary features are weighted using a cross-attention mechanism to obtain weighted semantic features and weighted boundary features. The weighted semantic features and weighted boundary features are decoded by a decoding module to obtain the transparent objects in the image. Specifically, mapping semantic features and boundary features to semantic queries and boundary queries, and mapping semantic-boundary fusion features to semantic keys and semantic values ​​corresponding to semantic queries, and boundary keys and boundary values ​​corresponding to boundary queries, includes: The semantic features, boundary features, and semantic-boundary fusion features that are uniformly transformed into C×H×W shapes are flattened into 2D semantic features. 2D boundary features 2D semantic-boundary fusion features ;in, , C is the number of channels; The 2D semantic features are processed using a pre-trained semantic mapping matrix. 2D semantic-boundary fusion features Perform mapping to obtain semantic queries semantic keys and semantic value : in, , , A pre-trained semantic mapping matrix for the mapping matrix; The 2D boundary features are processed using a pre-trained boundary mapping matrix. 2D semantic-boundary fusion features Perform mapping to obtain boundary queries Boundary keys and boundary values : in, , , This is a pre-trained boundary mapping matrix.

2. The transparent object detection method as described in claim 1, characterized in that, The boundary features are weighted using a cross-attention mechanism, specifically including: Boundary queries respectively Boundary keys and boundary values The dimensions are reshaped to H×w×C, and the reshaped boundary query is obtained. Reshaping the boundary key and reshaping boundary values ; exist In terms of dimensions, the boundary query will be reshaped respectively. and reshaping boundary keys Divided into Each region is divided into several regions, and average pooling is performed on each region to obtain a first weight representation matrix Q and a second weight representation matrix k. Based on the first weight representation matrix Q and the second weight representation matrix k, the region matching score matrix G is obtained. The superscript T indicates the transpose operation; In the region matching score matrix Select the top-ranked rows by score A fraction, The indices of the fractions are combined to obtain a matrix. And retrieve the reshaped boundary key based on the index. and reshaping boundary values The corresponding area in the middle: in, The torch.gather operation is used to extract elements from an input tensor along a specified dimension based on indices and construct a new tensor. and They are respectively with The key features and value features with the highest correlation; Will and Calculate spatial attention after transforming into a 2D shape : in, and They are respectively to and The 2D flattened bond features and 2D flattened value features obtained after transforming into a 2D shape; d = C / h, where h is the number of attention heads; 2D boundary features are connected via residual connections. Add spatial attention The calculation results yield weighted boundary features. : in, It is the element-wise addition operation in residual join.

3. The transparent object detection method as described in claim 2, characterized in that, The semantic features are weighted using a cross-attention mechanism, specifically including: Vs are weighted using a cross-attention mechanism to obtain an attention score. : Where d = C / h, h is the number of attention heads; the superscript T indicates the transpose operation; 2D semantic features are integrated through residual connections. Add attention score The calculation results yield weighted semantic features. : in, It is the element-wise addition operation in residual join.

4. The transparent object detection method as described in claim 3, characterized in that... The output of the cross-attention mechanism layer in the semantic-boundary co-evolution module is connected to a Mix-FFN network layer; the weighted semantic features and weighted boundary features output by the cross-attention mechanism layer are enhanced through the Mix-FFN network layer, specifically including: Where MPL() is the feedforward operation in the Mix-FFN network, Conv3×3() is the convolution operation in the Mix-FFN network, and GeLU() is the activation function operation in the Mix-FFN network; It is a weighted boundary feature. It enhances boundary features; It is a weighted semantic feature. It enhances semantic features.

5. The transparent object detection method as described in claim 1, characterized in that, The residual network module is formed by a ResNeXt-101 network; multiple residual convolutional layers of the ResNeXt-101 network are used to extract multi-scale features of transparent objects in the image to be detected. For i ∈ {0, 1, 2, 3, 4, 5}, the feature with the most semantic information is selected. As semantic features of transparent objects in the image to be detected.

6. The transparent object detection method as described in claim 5, characterized in that, The boundary detection module output and the residual network module output are connected to a boundary refinement module, which includes an aggregation layer, a residual layer, a spatial attention mechanism layer, and a fusion layer. The boundary refinement module refines the boundary features output by the boundary detection module, specifically including: The features output by the ResNeXt-101 network are aggregated through an aggregation layer. ,feature ,feature ,feature After upsampling to the same dimension, the features are aggregated with the boundary features: Upsample() is the upsampling operation. These are the boundary features extracted by the Canny edge detection algorithm in the boundary detection module; [] represents the feature aggregation operation; The upsampled feature map, which has the most boundary features but the least semantic information, is enhanced by the residual layer. : In this context, ReLU() is the activation function, BN() is the batch normalization operation, and Conv3×3() is the convolution operation. It is an enhanced boundary feature; It is the element-wise addition operation in residual join; It is the initial enhanced boundary feature ; The initial enhanced boundary features are obtained through a spatial attention mechanism layer. Further enhancements: Where Sigmoid() is the sigmoid function used for normalization. The attention map is obtained from the j-th upsampled feature map. This is an element-wise multiplication operation; This is the enhanced j-th boundary feature; The fourth boundary feature output by the spatial attention mechanism layer is incorporated through a fusion layer. The features with the most semantic information output by the ResNeXt-101 network Perform feature aggregation to obtain refined boundary features. : 。 7. The transparent object detection method as described in claim 6, characterized in that, The semantic-boundary co-evolution module includes a first semantic-boundary co-evolution module, a second semantic-boundary co-evolution module, a third semantic-boundary co-evolution module, and a fourth semantic-boundary co-evolution module connected in sequence; the first, second, third, and fourth semantic-boundary co-evolution modules enhance the semantic features and boundary features in the feature transfer process by being connected in series.

8. A transparent object detection device, characterized in that, include: A model building module is used to build a neural network, which includes a parallel residual network module and a boundary detection module, a semantic-boundary co-evolution module connected to the output of the residual network module and the boundary detection module, and a decoding module connected to the output of the semantic-boundary co-evolution module; wherein, the semantic-boundary co-evolution module includes a fusion layer and a cross-attention mechanism layer connected in sequence. The model training module is used to collect images containing transparent objects to build a dataset, and to train the neural network using the dataset to obtain a transparent object detection model. The detection module is used to input the image to be detected containing transparent objects into the detection model. It extracts semantic features of the transparent objects in the image through a residual network module; extracts boundary features of the transparent objects in the image through an edge detection algorithm in the boundary detection module; fuses semantic features and boundary features through a fusion layer in the semantic-boundary co-evolution module to obtain semantic-boundary fusion features of the transparent objects; maps semantic features and boundary features to semantic queries and boundary queries respectively through a cross-attention mechanism layer in the semantic-boundary co-evolution module, and maps the semantic-boundary fusion features to semantic keys and semantic values ​​corresponding to semantic queries and boundary keys and boundary values ​​corresponding to boundary queries. Based on the obtained queries, keys, and values, the semantic features and boundary features are weighted using a cross-attention mechanism to obtain weighted semantic features and weighted boundary features; and decodes the weighted semantic features and weighted boundary features through a decoding module to obtain the transparent objects in the image to be detected. Specifically, mapping semantic features and boundary features to semantic queries and boundary queries, and mapping semantic-boundary fusion features to semantic keys and semantic values ​​corresponding to semantic queries and boundary keys and boundary values ​​corresponding to boundary queries, includes: The semantic features, boundary features, and semantic-boundary fusion features that are uniformly transformed into C×H×W shapes are flattened into 2D semantic features. 2D boundary features 2D semantic-boundary fusion features ;in, , C is the number of channels; The 2D semantic features are processed using a pre-trained semantic mapping matrix. 2D semantic-boundary fusion features Perform mapping to obtain semantic queries semantic keys and semantic value : in, , , A pre-trained semantic mapping matrix for the mapping matrix; The 2D boundary features are processed using a pre-trained boundary mapping matrix. 2D semantic-boundary fusion features Perform mapping to obtain boundary queries Boundary keys and boundary values : in, , , This is a pre-trained boundary mapping matrix.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 7.

Images