Method and apparatus for detecting transparent objects, and non-transitory storage medium

Abstract

The application discloses a transparent object detection method and device and a nonvolatile storage medium. The method comprises the following steps: determining n layers of first feature images output by n layers of preset convolutional neural network models; performing first processing on the n-th layer of first feature images to obtain an initial positioning image of a transparent object; performing second processing on the n-1-th layer of first feature images and the initial positioning image to obtain a second feature image; performing third processing on the second feature image and the initial positioning image to obtain a third feature image; performing second processing and third processing on the corresponding feature images of each layer to obtain a first target feature image corresponding to the second layer; and performing third processing on the first feature image of the first layer and the first target feature image to obtain a detection result of the transparent object. The application solves the technical problem that the related art cannot accurately detect the transparent object due to insufficient and effective use of the feature images corresponding to the to-be-processed images of the transparent object.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for detecting transparent objects and a non-volatile storage medium. Background Technology

[0002] As the drone industry matures, accurately detecting transparent glass objects and performing obstacle avoidance maneuvers in real time during drone flight has become an important research topic. In recent years, salient object detection, as a binary image segmentation task, has developed rapidly, but glass object detection has not received sufficient attention. Glass differs from ordinary conspicuous objects; due to its unique optical properties of specular reflection and perspective, it is often indistinguishable even to the human eye. In the human visual system, when a target appears within the field of vision, people first locate the target in space and then observe the details of other parts. For example, transparent glass often reflects objects or refracted light again. Traditional salient object detection methods struggle to accurately locate glass-like objects, necessitating model improvements to capture richer global semantics and more detailed local image information.

[0003] Currently, similar recognition methods typically fall into the following categories: Early salient object detection methods utilized bottom-up models to predict salientity maps, such as contrast, boundary background, and center prior. Among these, multi-scale feature fusion is a commonly used approach that combines complementary features.

[0004] However, most of the segmentation methods mentioned above are not suitable for glass object detection tasks because glass areas typically occupy a large space in the input image. Furthermore, the strong difference between shadowed and unshadowed areas in glass images is not very obvious. Transparent glass can completely reveal objects behind it, making it difficult for models to detect and accurately segment glass from other objects. However, in recent years, convolutional neural networks have gradually shown their superiority in glass detection tasks. However, related convolutional neural network-based glass detection methods have overlooked an important point: they do not fully and effectively utilize the extracted features, resulting in less than ideal detected glass shapes, rough edges, inaccurate positions, and even false positives and false negatives.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a method and apparatus for detecting transparent objects, as well as a non-volatile storage medium, to at least solve the technical problem that the inability to accurately detect transparent objects is caused by the fact that related technologies do not fully and effectively utilize the feature images corresponding to the image to be processed of the transparent object.

[0007] According to one aspect of the embodiments of this application, a method for detecting transparent objects is provided, comprising: inputting an image of a transparent object to be processed into an n-layer preset convolutional neural network model to obtain an n-layer first feature image output by the n-layer preset convolutional neural network model, wherein n is a positive integer; performing a first processing on the first feature image of the nth layer in the n-layer first feature image to obtain an initial localization image of the transparent object, wherein the first processing includes: determining the position information of pixels in the image; performing a second processing on the first feature image of the (n-1)th layer and the initial localization image to obtain a second feature image, wherein the second processing includes: determining the semantic features of pixels; performing a third processing on the second feature image and the initial localization image to obtain a third feature image, wherein the third processing includes: performing multi-scale segmentation and fusion of the feature images; performing the second processing and the third processing sequentially on the feature images corresponding to each layer from the (n-2)th layer to the second layer to obtain a first target feature image corresponding to the second layer; and performing a third processing on the first feature image of the first layer and the first target feature image to obtain a detection result of the transparent object.

[0008] Optionally, the first feature image of the nth layer in the first feature image of the nth layer is input into the global perception module of the nth layer to obtain the initial localization image of the transparent object, wherein the global perception module is used to determine the position information of pixels in the image; the first feature image of the (n-1)th layer and the initial localization image are input into the global functional polishing network module of the (n-1)th layer to obtain the second feature image, wherein the global functional polishing network module is used to determine the semantic features of pixels; the second feature image and the initial localization image are input into the multi-scale thinning module of the (n-1)th layer to obtain the third feature image, wherein the multi-scale thinning module is used to perform multi-scale segmentation and fusion of the feature image; from the (n-2)th layer to the 2nd layer, the image output by the global functional polishing network module of each layer and the image output by the multi-scale thinning module of the previous layer are sequentially input into the multi-scale thinning module of the current layer to obtain the first target feature image output by the multi-scale thinning module of the 2nd layer; the first feature image of the 1st layer and the first target feature image are input into the multi-scale thinning module of the 1st layer to obtain the detection result of the transparent object.

[0009] Optionally, the first feature image of the nth layer in the first feature image of the nth layer is input into the global perception module of the nth layer to obtain the initial localization image of the transparent object, including: converting the first feature in the first feature image of the nth layer into the second feature and the third feature to obtain the first image, wherein the first feature includes: the number of channels of the image, the height of the image, and the width of the image; the second feature includes: the number of channels of the image and the height of the image; and the third feature includes: the number of channels of the image and the width of the image; performing convolution and expansion processing on the first image to obtain the second image; summing the second feature and the third feature in the second image to obtain the third image; performing convolution and normalization processing on the third image to obtain the fourth image; and multiplying the pixel values ​​corresponding to the fourth image and the image to be processed to obtain the initial localization image.

[0010] Optionally, the first feature image of layer n-1 and the initial localization image are input into the global function polishing network module of layer n-1 to obtain the second feature image, including: performing two different convolution processes on the initial localization image to obtain the fifth image and the sixth image; transposing the first matrix corresponding to the seventh image to obtain the first target matrix, wherein the seventh image is any one of the fifth image and the sixth image; multiplying the first target matrix and the second matrix to obtain the second target matrix, wherein the second matrix is ​​the matrix corresponding to the eighth image, wherein the eighth image is any one of the fifth image and the sixth image except the seventh image; activating the second target matrix using a preset activation function to obtain the third target matrix; performing convolution processing on the first feature image of layer n-1 to obtain the ninth image; and determining the second feature image based on the third matrix corresponding to the ninth image and the third target matrix.

[0011] Optionally, before inputting the second feature image and the initial positioning image into the multi-scale thinning module of the (n-1)th layer to obtain the third feature image, the method further includes: dividing the second feature image into x groups according to the number of channels of the multi-scale thinning module to obtain the tenth image of x groups.

[0012] Optionally, the second feature image and the initial localization image are input into the multi-scale thinning module of the (n-1)th layer to obtain the third feature image, including: stitching the first group of tenth images in x groups with the initial localization image to obtain the eleventh image; using the eleventh image as the input image of the second branch from the first branch, where the first branch is the branch containing the first group of tenth images and the second branch is the branch containing the second group of tenth images; performing convolution processing on the eleventh image to obtain the first target image; stitching the tenth images with the input images from the second group to the xth group to obtain x-1 output images; performing convolution processing on each of the x-1 output images to obtain x-1 target images; and determining the third feature image based on the first target image and the x-1 target images.

[0013] Optionally, determining a third feature image based on the first target image and x-1 target images includes: summing the first target image and x-1 target images to obtain a second target image; performing convolution processing on the second target image to obtain a third target image; and summing the third target image and the initial positioning image to obtain the third feature image.

[0014] According to another aspect of the embodiments of this application, a device for detecting transparent objects is also provided, comprising: an extraction module, configured to input an image of the transparent object to be processed into an n-layer preset convolutional neural network model to obtain an n-layer first feature image output by the n-layer preset convolutional neural network model, wherein n is a positive integer; a first processing module, configured to perform a first processing on the first feature image of the nth layer in the n-layer first feature image to obtain an initial positioning image of the transparent object, wherein the first processing includes: determining the position information of pixels in the image; and a second processing module, configured to perform a second processing on the first feature image of the (n-1)th layer and the initial positioning image. The second processing module is used to obtain a second feature image, wherein the second processing includes: determining the semantic features of pixels; a third processing module is used to perform a third processing on the second feature image and the initial localization image to obtain a third feature image, wherein the third processing includes: multi-scale segmentation and fusion of the feature image; a fourth processing module is used to perform the second processing and the third processing on the feature images corresponding to each layer from the (n-2)th layer to the 2nd layer in sequence to obtain the first target feature image corresponding to the 2nd layer; a fifth processing module is used to perform a third processing on the first feature image and the first target feature image of the 1st layer to obtain the detection result of the transparent object.

[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the storage medium including a stored program, wherein the program, when running, controls the device where the storage medium is located to execute the above-described method for detecting transparent objects.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes the above-described method for detecting transparent objects.

[0017] In this embodiment, the image of the transparent object to be processed is input into an n-layer preset convolutional neural network model to obtain an n-layer first feature image output by the n-layer preset convolutional neural network model, where n is a positive integer; a first processing is performed on the first feature image of the nth layer in the n-layer first feature image to obtain an initial positioning image of the transparent object, wherein the first processing includes: determining the position information of pixels in the image; a second processing is performed on the first feature image of the (n-1)th layer and the initial positioning image to obtain a second feature image, wherein the second processing includes: determining the semantic features of pixels; a third processing is performed on the second feature image and the initial positioning image to obtain a third feature image, wherein the third processing includes: The feature image is segmented and fused at multiple scales. From the (n-2)th layer to the 2nd layer, the feature images corresponding to each layer are processed in the second and third steps to obtain the first target feature image corresponding to the 2nd layer. The first feature image and the first target feature image of the 1st layer are processed in the third step to obtain the detection result of the transparent object. By processing the first feature image multiple times, the feature image corresponding to the image to be processed of the transparent object is fully and effectively utilized, thereby achieving the technical effect of accurate detection of transparent objects. This solves the technical problem that the related technologies cannot accurately detect transparent objects because they do not fully and effectively utilize the feature image corresponding to the image to be processed of the transparent object. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a flowchart of a method for detecting a transparent object according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of a method for detecting a transparent object according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the structure of a global function polishing network module according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the structure of a multi-scale refinement module according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of an experimental result according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of a visual comparison according to an embodiment of this application;

[0025] Figure 7 This is a schematic diagram illustrating another experimental result according to an embodiment of this application;

[0026] Figure 8 This is a structural diagram of a transparent object detection device according to an embodiment of this application;

[0027] Figure 9 This is a hardware structure block diagram of a computer terminal (or electronic device) for a method of detecting transparent objects according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] According to an embodiment of this application, a method embodiment for detecting transparent objects is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] Figure 1 This is a flowchart of a method for detecting transparent objects according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0032] Step S102: Input the image of the transparent object to be processed into an n-layer preset convolutional neural network model to obtain the n-layer first feature image output by the n-layer preset convolutional neural network model, where n is a positive integer.

[0033] Step S104: Perform a first processing on the first feature image of the nth layer in the first feature image of the nth layer to obtain the initial positioning image of the transparent object. The first processing includes: determining the position information of pixels in the image.

[0034] Step S106: Perform a second processing on the first feature image and the initial localization image of the (n-1)th layer to obtain a second feature image, wherein the second processing includes: determining the semantic features of the pixels.

[0035] Step S108: Perform a third processing on the second feature image and the initial localization image to obtain a third feature image. The third processing includes: performing multi-scale segmentation and fusion on the feature image.

[0036] Step S110: From the (n-2)th layer to the 2nd layer, perform the second and third processing on the feature images corresponding to each layer in sequence to obtain the first target feature image corresponding to the 2nd layer.

[0037] Step S112: Perform third processing on the first feature image and the first target feature image of the first layer to obtain the detection result of the transparent object.

[0038] Based on the above steps, by processing the first feature image multiple times, the goal of fully and effectively utilizing the feature image corresponding to the image to be processed of the transparent object is achieved, thereby realizing the technical effect of accurately detecting transparent objects.

[0039] This application also provides a method for detecting transparent objects, including the following steps:

[0040] Step S202: Input the image of the transparent object to be processed into an n-layer preset convolutional neural network model to obtain the n-layer first feature image output by the n-layer preset convolutional neural network model.

[0041] Figure 2 This is a schematic diagram of a method for detecting transparent objects according to an embodiment of this application, as shown below. Figure 2 As shown, the image to be processed is input into a 5-layer preset convolutional neural network model ResNeXt, and the first feature image of the 5 layers output by the 5-layer preset convolutional neural network model is obtained, namely X→{X1,X2,X3,X4,X5}.

[0042] Figure 2 It includes the following modules, among which,

[0043] The Global Perception (GP) module is used to insert multiple strip pools in a stacked manner at the top layer of the convolutional neural network backbone, thereby effectively capturing the features of long and narrow targets.

[0044] This diagram illustrates the structure of the Global Feature Polishing (GFP) network module. GFP is used to introduce high-confidence semantic information into the deep guidance network to capture relevant features of long-distance pixel semantics. Glass objects typically have continuous and regular boundaries. During the upsampling process of a fully convolutional neural network, the mapping relationship between pixels is gradually diluted, making it impossible to retain long-distance dependencies between pixels, ultimately leading to missing boundaries and incorrect detection results. To address these issues, this embodiment provides a Global Feature Polishing network module to obtain long-distance dependencies between different pixels.

[0045] The Multi-scale Refinement (MSR) module is designed to fully preserve high-level semantic information while filtering out noise interference from shallow layers as much as possible. After obtaining the initial coarse localization map, the model needs to integrate multi-level convolutional features to compensate for missing target parts and correct erroneous detections. Different levels of deep convolutional neural networks learn features for different scales: shallow layers capture low-level structural information, while deep layers capture high-level semantic information. This module is added to the model to attempt to effectively integrate their complementary information to obtain better feature representations. Generally, seamless aggregation has two basic requirements: fully preserving high-level semantic information while filtering out noise interference from shallow layers as much as possible. Based on these two principles, the model designs an efficient guided refinement module, namely the multi-scale refinement module, through feature fusion.

[0046] For the fifth layer, the input to GP is X5, and the output of GP is the initial localization image y. For the fourth layer, the input to GFP is X4 and y, and the output of GFP is the second feature image. The input to MSR is the second feature image. And y. For the third layer, the input of GFP is X3 and the output of the fourth layer MSR, the input of MSR is the output of the fourth layer MSR and the third layer GFP, and so on.

[0047] Specifically, to input the image to be processed into a 5-layer convolutional neural network model, the following steps can be taken:

[0048] Step S2021: Define the structure of the convolutional neural network model, including convolutional layers, pooling layers, and fully connected layers. The specific network structure can be designed according to task requirements and data characteristics.

[0049] Step S2022: Input the image to be processed into the first convolutional layer. The convolutional layer uses a convolution kernel to perform a convolution operation on the input image to generate the first feature image X1;

[0050] Step S2023: Input the first layer feature image into the second convolutional layer. Similarly, the second convolutional layer performs a convolution operation on the first layer feature image to generate the second layer feature image X2;

[0051] Step S2024: Repeat step 2023, and input the second layer feature image into the third, fourth and fifth convolutional layers in sequence to generate the corresponding feature images X3, X4 and X5.

[0052] Understandably, an activation function (such as ReLU) can be added after each convolutional layer to introduce non-linearity.

[0053] Step S204: Input the first feature image of the nth layer from the first feature image of the nth layer into the global perception module of the nth layer to obtain the initial positioning image of the transparent object. The global perception module is used to determine the position information of pixels in the image.

[0054] The specific implementation method of step S204 is as follows:

[0055] The first feature in the first feature image of the nth layer is transformed into the second and third features to obtain the first image. The first feature includes the number of channels, the height, and the width of the image. The second feature includes the number of channels and the height of the image. The third feature includes the number of channels and the width of the image. The first image is convolved and expanded to obtain the second image. The second and third features in the second image are summed to obtain the third image. The third image is convolved and normalized to obtain the fourth image. The pixel values ​​of the fourth image and the image to be processed are multiplied to obtain the initial localization image.

[0056] For example, the first feature image X5 of the 5th layer can be represented as C×H×W (first feature), where C, H, and W are the number of channels, height, and width of the image, respectively. After horizontal and vertical strip pooling, X5 is transformed into the first image, which includes the second and third features, namely H×C and C×W. The first image is input into a convolutional layer with a kernel of 3 and expanded to obtain the second image. At the corresponding positions, the third features of the second and third features are summed to obtain the third image, which includes the following feature: H×W. The third image is convolved with C×C and normalized to obtain the fourth image. The fourth image is multiplied by the pixel values ​​corresponding to the image X to be processed to obtain the output result, i.e., the initial localization image y.

[0057] Step S206: Input the first feature image and the initial localization image of the (n-1)th layer into the global function polishing network module of the (n-1)th layer to obtain the second feature image, wherein the global function polishing network module is used to determine the semantic features of the pixels.

[0058] Figure 3 This is a schematic diagram of the Global Feature Polishing (GFP) network module. GFP is used to guide the introduction of high-confidence semantic information into the deep network to capture relevant features of long-distance pixel semantics.

[0059] The specific implementation method of step S206 is as follows: Perform two different convolution processes on the initial positioning image to obtain the fifth image and the sixth image; transpose the first matrix corresponding to the seventh image to obtain the first target matrix, where the seventh image is any one of the fifth and sixth images; multiply the first target matrix and the second matrix to obtain the second target matrix, where the second matrix is ​​the matrix corresponding to the eighth image, and the eighth image is any one of the fifth and sixth images except the seventh image; activate the second target matrix using a preset activation function to obtain the third target matrix; convolve the first feature image of the (n-1)th layer to obtain the ninth image; determine the second feature image based on the third matrix and the third target matrix corresponding to the ninth image.

[0060] For example, the output of the previous layer, the initial localization image y, is input into two 1×1 convolutional layers to obtain two new feature maps: the fifth image and the sixth image. The matrix K corresponding to one of the fifth and sixth images is transposed to obtain K. T (First target matrix), the matrix corresponding to the other image in the fifth and sixth images is denoted as Q (second matrix). For K T Multiplying Q by Q yields the second objective matrix. Further, activating the second objective matrix using a preset activation function yields the third objective matrix.

[0061] In summary, the third objective matrix σ is determined. j,i The process can be represented by the following formula:

[0062]

[0063] Where, σ j,i This represents the dependency between positions i and j in the image, where i and j are the coordinates in the image; T is the matrix transpose symbol.

[0064] Furthermore, feature extraction is performed on the (n-1)th first feature image, which means inputting X4 into another 1×1 convolutional layer to obtain the ninth image. The matrix (third matrix) corresponding to the ninth image is denoted as V.

[0065] Based on the third matrix V and the third objective matrix σ j,i The second feature image is determined, and the process of determining the second feature image can be represented by the following formula:

[0066]

[0067] Where α is the scale parameter. In this embodiment, the scale parameter is initialized to 0 and more weights are gradually allocated during the training process.

[0068] Understandably, traditional non-local modules rely solely on input features to explore self-attention relationships within the module. In contrast, GFP introduces high-confidence semantic information, guiding the network at a deeper level to capture relevant features of long-distance pixel semantics. This effectively improves feature representation and facilitates subsequent refinement, resulting in more accurate detection results.

[0069] Step S208: Input the second feature image and the initial localization image into the multi-scale thinning module of the (n-1)th layer to obtain the third feature image. The multi-scale thinning module is used to perform multi-scale segmentation and fusion of the feature image.

[0070] The specific implementation method of step S208 is as follows: Based on the number of channels of the multi-scale thinning module, the second feature image is divided into x groups to obtain x groups of tenth images. The tenth image of the first group in the x groups of tenth images is stitched with the initial positioning image to obtain the eleventh image. The eleventh image is used as the input image of the second branch from the first branch, where the first branch is the branch where the tenth image of the first group is located, and the second branch is the branch where the tenth image of the second group is located. The eleventh image is convolved to obtain the first target image. From the second group to the xth group, the tenth image is stitched with the input image to obtain x-1 output images. The x-1 output images are convolved to obtain x-1 target images.

[0071] The first target image and x-1 target images are summed to obtain the second target image; the second target image is convolved to obtain the third target image; the third target image and the initial localization image are summed to obtain the third feature image.

[0072] For example, let C be the number of channels in the multi-scale refinement module, and then... Before inputting into the multi-scale refinement module, Divide the data into g groups, each group consisting of C / g channel feature maps. Assume that... Divide into 4 groups, and get The input is a multi-scale refinement module with 4 branches. Further, the initial localization image y is used as a guide and connected to the C / g channel feature maps within each group, resulting in (C / g+1) channels.

[0073] The multi-scale refinement module selects embedding convolutions after concatenating the feature maps in all branches except the last one to further increase the receptive field. Furthermore, starting from the second branch, it uses the output of the previous branch as guidance to concatenate with the input feature map of the current branch to further preserve the feature information in the current branch.

[0074] The multi-scale refinement module is specifically represented as: By concatenating the image with the initial localization image y, we obtain... Will As the input image from the first branch to the second branch, for the first... Perform convolution processing to obtain

[0075] In the second branch, and After splicing, the following is obtained: right Perform convolution processing to obtain

[0076] In the third branch, and After splicing, the following is obtained: right Perform convolution processing to obtain

[0077] In the fourth branch, and After splicing, the following is obtained: right Perform convolution processing to obtain

[0078] In summary, and By inputting a multi-scale refinement module with 4 branches, we can obtain... and

[0079] Furthermore, and The second target image is obtained by summing the results; the third target image is obtained by convolution of the second target image; the third feature image is obtained by summing the third target image and the initial localization image y.

[0080] Step S210: Based on step S208, from layer n-2 to layer 2, the images output by the global function polishing network module of each layer and the images output by the multi-scale thinning module of the previous layer are sequentially input into the multi-scale thinning module of the current layer to obtain the first target feature image output by the multi-scale thinning module of the second layer.

[0081] For example, when n=5, taking the 3rd layer as an example, the input of the global function polishing network module of the 3rd layer is: X3 and the output of the multi-scale refinement module of the 4th layer; the input of the multi-scale refinement module of the 3rd layer is: the output of the global function polishing network module of the 3rd layer and the output of the multi-scale refinement module of the 4th layer.

[0082] Step S212: Input the first feature image and the first target feature image of the first layer into the multi-scale thinning module of the first layer to obtain the detection result of the transparent object.

[0083] To evaluate the effectiveness of the transparent object detection method provided in this application, a test dataset was used to test the method. The test dataset includes GDD data and Window data. GDD is the first large-scale benchmark dataset in the image processing field specifically for glass detection, containing 3916 glass images. Window is a recent dataset compiled in this application for model training, containing 4419 glass images, including large glass images of different categories, mainly including windows, glass walls, glass doors, and other materials with different angles and transparency. All images in Window were selected from complex real-world scenes with significant differences in physical scale, viewpoint, contrast, occlusion, category, and transparency, thus meeting the needs of model training and testing.

[0084] This method was tested using four widely used metrics: F-metric (Fβ), E-metric (Eφ), S-metric (Sα), and MAE (M). In this embodiment, the threshold for F is adaptive, set to twice the average value. Only the maximum Fβ and Eφ are reported here to show the best performance a model can achieve. Higher values ​​for Fβ, Eφ, and Sα indicate better model performance, while the opposite is true for M.

[0085] The implementation details for evaluating the effectiveness of the transparent object detection method provided in this application are as follows:

[0086] First, it should be noted that all experiments were implemented on PyTorch, using an NVIDIA XP GPU and an Intel i7-6700K CPU 4.0GHz processor. The ResNeXt-101 pre-trained dataset on ImageNet was used as the backbone. Data augmentation methods such as random flipping and multi-scale input images were applied to mitigate the risk of overfitting. All images were resized to 352×352 for both training and testing. The training set was the same as that of GDNet. To obtain high-quality region segmentation and sharp boundaries, the hybrid loss was defined as the sum of the Intersection over Union (IOU) loss and the Binary Cross Entropy (BCE) loss. The hyperparameter settings were as follows: initial learning rate of 0.0005, decreasing by 10 after 25 epochs; batch size of 10; and maximum epochs of 50.

[0087] This application is compared with state-of-the-art (SOTA) models: This application is compared with other state-of-the-art methods. Figure 5 Detailed experimental results are shown, demonstrating that this application significantly outperforms other methods on the GOD dataset, particularly in Eφ and Sα, achieving a 0.47% improvement over the second method. This application also achieved the highest evaluation score for windowing.

[0088] Figure 6 A visualization showing the comparison results between the method provided in this application and other advanced methods is presented. Figure 6 It's intuitively clear that the relevant methods perform poorly when glass occupies a large portion of the image. For Spatially Constrained Region-based Convolutional Neural Networks (SCRN), boundary supervision is ineffective in this task. When the object is glass rather than a significant object (rows 3 and 4), Figure 6 Most of the salient object detection methods listed above make errors in object localization. Results show that even after a glass object is accurately located, the object boundary predicted by the network still exhibits some distortion and blurring. In contrast, the method presented in this application achieves impressive results, demonstrating the effectiveness of the proposed module.

[0089] To demonstrate that the global feature polishing module proposed in this application has a significant advantage over the classic self-attention module, this application attempts to integrate the two modules into the baseline for comparison. The results can be seen from the tabs. Figure 7The proposed GFP model significantly improves various performance metrics, particularly the s-metric, which is 1.2% higher than the baseline. However, when non-local modules are replaced with the proposed GFP, performance degrades significantly, with the s-metric decreasing from 0.824 to 0.821, and Eφ and Fβ also decreasing by 0.7% and 0.5%, respectively. This indicates that optimizing the input features of each layer can significantly improve the performance of the final output. The GFP proposed in this application greatly enhances the network's ability to extract spatial structure features, which is beneficial for current glass detection tasks.

[0090] Figure 8 This is a structural diagram of a transparent object detection device according to an embodiment of this application, as shown below. Figure 8 As shown, the device includes:

[0091] The extraction module 81 is used to input the image of the transparent object to be processed into an n-layer preset convolutional neural network model to obtain the first feature image of the n-layer preset convolutional neural network model, where n is a positive integer;

[0092] The first processing module 82 is used to perform a first processing on the first feature image of the nth layer in the nth layer first feature image to obtain the initial positioning image of the transparent object, wherein the first processing includes: determining the position information of pixels in the image;

[0093] The second processing module 83 is used to perform a second processing on the first feature image and the initial positioning image of the (n-1)th layer to obtain a second feature image, wherein the second processing includes: determining the semantic features of the pixels;

[0094] The third processing module 84 is used to perform third processing on the second feature image and the initial positioning image to obtain a third feature image. The third processing includes: performing multi-scale segmentation and fusion on the feature image.

[0095] The fourth processing module 85 is used to perform second and third processing on the feature images corresponding to each layer from the (n-2)th layer to the 2nd layer in sequence to obtain the first target feature image corresponding to the 2nd layer;

[0096] The fifth processing module 86 is used to perform third processing on the first feature image and the first target feature image of the first layer to obtain the detection result of the transparent object.

[0097] It should be noted that the above Figure 8 The modules in the above can be program modules (e.g., a set of program instructions that implement a specific function) or hardware modules. For the latter, they can be represented in the following forms, but are not limited to these: each of the above modules is represented by a processor, or the functions of each of the above modules are implemented by a processor.

[0098] It should be noted that, Figure 8 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 1 The relevant descriptions of the embodiments shown will not be repeated here.

[0099] Figure 9 A hardware block diagram of a computer terminal (or mobile device) for implementing a method for detecting transparent objects is shown. Figure 9 As shown, a computer terminal 90 (or mobile device) may include one or more processors 902 (shown as 902a, 902b, ..., 902n in the figure) 902 (processor 902 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 904 for storing data, and a transmission module 906 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 9 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the computer terminal 90 may also include... Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.

[0100] It should be noted that the aforementioned one or more processors 902 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 90 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0101] The memory 904 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the transparent object detection method in this embodiment. The processor 902 executes various functional applications and data processing by running the software programs and modules stored in the memory 904, thereby realizing the aforementioned transparent object detection method. The memory 904 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 904 may further include memory remotely located relative to the processor 902, and these remote memories can be connected to the computer terminal 90 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0102] The transmission module 906 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 90. In one example, the transmission module 906 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 906 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0103] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of a computer terminal 90 (or a mobile device).

[0104] It should be noted here that, in some optional embodiments, the above... Figure 9 The computer device (or electronic device) shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 9 This is only one instance of a particular specific instance, and is intended to illustrate the types of components that may exist in the aforementioned computer equipment (or electronic equipment).

[0105] It should be noted that, Figure 9 The electronic device shown is used to perform Figure 1 The method for detecting transparent objects shown above also applies to this electronic device, and will not be repeated here.

[0106] This application also provides a non-volatile storage medium, which includes a stored program, wherein the program, when running, controls the device where the storage medium is located to execute the above-described method for detecting transparent objects.

[0107] A non-volatile storage medium performs the following functions: It inputs an image of a transparent object into an n-layer pre-defined convolutional neural network model to obtain an n-layer first feature image output by the model, where n is a positive integer; it performs a first processing on the first feature image of the nth layer to obtain an initial localization image of the transparent object, wherein the first processing includes determining the positional information of pixels in the image; it performs a second processing on the first feature image of the (n-1)th layer and the initial localization image to obtain a second feature image, wherein the second processing includes determining the semantic features of pixels; it performs a third processing on the second feature image and the initial localization image to obtain a third feature image, wherein the third processing includes multi-scale segmentation and fusion of the feature images; from the (n-2)th layer to the 2nd layer, it sequentially performs the second and third processing on the feature images corresponding to each layer to obtain a first target feature image corresponding to the 2nd layer; it performs a third processing on the first feature image of the 1st layer and the first target feature image to obtain the detection result of the transparent object.

[0108] This application also provides an electronic device, including a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the above-described method for detecting transparent objects.

[0109] The processor is used to run a program that performs the following functions: inputting the image of the transparent object to be processed into an n-layer preset convolutional neural network model to obtain an n-layer first feature image output by the n-layer preset convolutional neural network model, where n is a positive integer; performing a first processing on the first feature image of the nth layer in the n-layer first feature image to obtain an initial localization image of the transparent object, wherein the first processing includes: determining the position information of pixels in the image; performing a second processing on the first feature image of the (n-1)th layer and the initial localization image to obtain a second feature image, wherein the second processing includes: determining the semantic features of pixels; performing a third processing on the second feature image and the initial localization image to obtain a third feature image, wherein the third processing includes: performing multi-scale segmentation and fusion of the feature images; performing the second and third processing on the feature images corresponding to each layer from the (n-2)th layer to the 2nd layer in sequence to obtain a first target feature image corresponding to the 2nd layer; performing a third processing on the first feature image of the 1st layer and the first target feature image to obtain the detection result of the transparent object.

[0110] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0111] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0116] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for detecting transparent objects, characterized in that, include: The image of the transparent object to be processed is input into an n-layer pre-defined convolutional neural network model to obtain the first feature image of the n-layer pre-defined convolutional neural network model, where n is a positive integer; The first feature image of the nth layer in the nth layer of the first feature image is processed to obtain the initial positioning image of the transparent object, wherein the first processing includes: determining the position information of pixels in the image; The first feature image of the (n-1)th layer and the initial positioning image are subjected to a second processing to obtain a second feature image, wherein the second processing includes: determining the semantic features of the pixels; The second feature image and the initial positioning image are subjected to a third processing to obtain a third feature image, wherein the third processing includes: multi-scale segmentation and fusion of the feature image; From the (n-2)th layer to the 2nd layer, the second processing and the third processing are performed sequentially on the feature images corresponding to each layer to obtain the first target feature image corresponding to the 2nd layer; The third processing is performed on the first feature image of the first layer and the first target feature image to obtain the detection result of the transparent object; The first feature image of the nth layer in the nth layer is input into the global perception module of the nth layer to obtain the initial localization image of the transparent object, wherein the global perception module is used to determine the position information of pixels in the image; the first feature image of the (n-1)th layer and the initial localization image are input into the global functional polishing network module of the (n-1)th layer to obtain the second feature image, wherein the global functional polishing network module is used to determine the semantic features of pixels; the second feature image and the initial localization image are input into the multi-scale thinning module of the (n-1)th layer to obtain the third feature image, wherein the multi-scale thinning module is used to perform multi-scale segmentation and fusion of the feature image; from the (n-2)th layer to the 2nd layer, the image output by the global functional polishing network module of each layer and the image output by the multi-scale thinning module of the previous layer are sequentially input into the multi-scale thinning module of the current layer to obtain the first target feature image output by the multi-scale thinning module of the 2nd layer; the first feature image of the 1st layer and the first target feature image are input into the multi-scale thinning module of the 1st layer to obtain the detection result of the transparent object; The process involves inputting the first feature image of the nth layer from the first feature image of the nth layer into the global perception module of the nth layer to obtain the initial positioning image of the transparent object. This includes: converting the first feature in the first feature image of the nth layer into a second feature and a third feature to obtain a first image, wherein the first feature includes the number of channels, the height, and the width of the image; the second feature includes the number of channels and the height of the image; and the third feature includes the number of channels and the width of the image. The first image is then convolved and expanded to obtain a second image. The second and third features in the second image are summed to obtain a third image. The third image is then convolved and normalized to obtain a fourth image. Finally, the pixel values ​​corresponding to the fourth image and the image to be processed are multiplied to obtain the initial positioning image.

2. The method according to claim 1, characterized in that, The first feature image of layer n-1 and the initial localization image are input into the global function polishing network module of layer n-1 to obtain the second feature image, including: The initial positioning image is subjected to two different convolution processes to obtain the fifth and sixth images; The first matrix corresponding to the seventh image is transposed to obtain the first target matrix, wherein the seventh image is any one of the fifth image and the sixth image; The first target matrix and the second matrix are multiplied to obtain the second target matrix, wherein the second matrix is ​​the matrix corresponding to the eighth image, and the eighth image is the fifth image and the sixth image excluding the seventh image; The second target matrix is ​​activated using a preset activation function to obtain the third target matrix; The first feature image of the (n-1)th layer is convolved to obtain the ninth image; The second feature image is determined based on the third matrix corresponding to the ninth image and the third target matrix.

3. The method according to claim 1, characterized in that, Before inputting the second feature image and the initial positioning image into the multi-scale thinning module of the (n-1)th layer to obtain the third feature image, the method further includes: dividing the second feature image into x groups according to the number of channels of the multi-scale thinning module to obtain the tenth image of x groups.

4. The method according to claim 3, characterized in that, The second feature image and the initial localization image are input into the multi-scale thinning module of the (n-1)th layer to obtain the third feature image, including: The first group of ten images in the x groups is stitched together with the initial positioning image to obtain the eleventh image. The eleventh image is used as the input image of the second branch from the first branch, wherein the first branch is the branch where the first group of ten images is located, and the second branch is the branch where the second group of ten images is located. The eleventh image is convolved to obtain the first target image; From the second group to the xth group, the tenth image is concatenated with the input image to obtain x-1 output images. Each of the x-1 output images is then convolved to obtain x-1 target images. A third feature image is determined based on the first target image and the x-1 target images.

5. The method according to claim 4, characterized in that, Based on the first target image and the x-1 target images, a third feature image is determined, including: The first target image and the x-1 target images are summed to obtain the second target image; The second target image is convolved to obtain the third target image; The third target image and the initial positioning image are summed to obtain the third feature image.

6. A device for detecting transparent objects, characterized in that, include: The extraction module is used to input the image of the transparent object to be processed into an n-layer preset convolutional neural network model to obtain the first feature image of the n-layer preset convolutional neural network model, where n is a positive integer; The first processing module is used to perform a first processing on the first feature image of the nth layer in the nth layer first feature image to obtain an initial positioning image of the transparent object, wherein the first processing includes: determining the position information of pixels in the image; The second processing module is used to perform a second processing on the first feature image of the (n-1)th layer and the initial positioning image to obtain a second feature image, wherein the second processing includes: determining the semantic features of the pixels; The third processing module is used to perform a third processing on the second feature image and the initial positioning image to obtain a third feature image, wherein the third processing includes: performing multi-scale segmentation and fusion on the feature image; The fourth processing module is used to sequentially perform the second processing and the third processing on the feature images corresponding to each layer from the (n-2)th layer to the 2nd layer to obtain the first target feature image corresponding to the 2nd layer; The fifth processing module is used to perform the third processing on the first feature image of the first layer and the first target feature image to obtain the detection result of the transparent object; The transparent object detection device is further configured to perform the following steps: inputting the first feature image of the nth layer from the nth layer first feature image into the global perception module of the nth layer to obtain an initial localization image of the transparent object, wherein the global perception module is used to determine the position information of pixels in the image; inputting the first feature image of the (n-1)th layer and the initial localization image into the global functional polishing network module of the (n-1)th layer to obtain a second feature image, wherein the global functional polishing network module is used to determine the semantic features of pixels; inputting the second feature image and the initial localization image into the multi-scale thinning module of the (n-1)th layer to obtain a third feature image, wherein the multi-scale thinning module is used to perform multi-scale segmentation and fusion of the feature image; from the (n-2)th layer to the 2nd layer, sequentially inputting the image output by the global functional polishing network module of each layer and the image output by the multi-scale thinning module of the previous layer into the multi-scale thinning module of the current layer to obtain the first target feature image output by the multi-scale thinning module of the 2nd layer; inputting the first feature image of the 1st layer and the first target feature image into the multi-scale thinning module of the 1st layer to obtain the detection result of the transparent object; The transparent object detection device is further configured to perform the following steps: converting a first feature in the first feature image of the nth layer into a second feature and a third feature to obtain a first image, wherein the first feature includes: the number of channels of the image, the height of the image, and the width of the image; the second feature includes: the number of channels of the image and the height of the image; and the third feature includes: the number of channels of the image and the width of the image; performing convolution and expansion processing on the first image to obtain a second image; summing the second feature and the third feature in the second image to obtain a third image; performing convolution and normalization processing on the third image to obtain a fourth image; and multiplying the pixel values ​​corresponding to the fourth image and the image to be processed to obtain the initial positioning image.

7. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the non-volatile storage medium to perform the method for detecting transparent objects as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the method for detecting a transparent object as described in any one of claims 1 to 5.

Images