A method for detecting salient targets of lightweight leather surface defects by introducing RBU

Abstract

The present invention proposes a method for detecting significant targets of lightweight leather surface defects by introducing RBU, obtaining a leather surface image and preprocessing it; using the model RBU 2 Net performs target detection on the preprocessed image. The present invention proposes a residual bottleneck U-block as a feature extraction unit, which combines the performance advantages of deep separable convolution and channel attention, and to a certain extent solves the dilemma of limited computing equipment in automated leather defect detection on industrial production lines. By introducing a neighboring feature enhancement module into the model, the model's ability to fuse multi-scale features is enhanced, and the current situation in the field of leather detection where the shape of defects cannot be accurately portrayed is optimized. In addition, pixel hybrid sampling is introduced to improve the utilization rate of feature information and enhance the spatial perception ability of the model, so that it can improve the accuracy of leather detection when detecting leather surface defects with a wide variety of types, irregular defect shapes, and a large range of scale variations.

Description

Technical Field

[0001] The invention relates to the technical field of surface defect detection and significant target detection, and in particular to a method for detecting significant targets of lightweight leather surface defects by introducing RBU. Background Art

[0002] Defect detection on the leather surface is an important prerequisite for improving product qualification rate and ensuring product quality, and it is a challenging practical application task. For a long time, leather defect detection has relied on manual work, and this traditional method has great limitations. First, manual inspection is easily affected by subjective factors and individual differences, resulting in unstable detection accuracy. Secondly, with the expansion of production scale and the increase of production speed, the efficiency of manual inspection has gradually become a bottleneck restricting the production process, limiting the development potential of the leather industry. In order to overcome these problems, automation technology based on deep learning has been gradually introduced in recent years to achieve more intelligent and efficient leather production.

[0003] The surface defects of leather not only have the characteristics of a wide variety, irregular defect shapes, and a large range of scale variations, but also the background texture of leather is complex and changeable, and the color is diverse, so the model is required to have good representation capabilities. Most of the existing leather defect detection technologies based on deep learning use target detection methods, which only aim to generate defect bounding boxes and cannot accurately characterize the shape of defects, so it is necessary to study defect detection methods with finer detection granularity. The salient object detection method clearly distinguishes the foreground and background in a pixel binary classification manner, which coincides with the goal of leather surface defect detection tasks. In the field of medical image detection, salient object detection methods have been relatively mature, but limited by the quality of public datasets and network model performance, the existing salient object detection methods are difficult to meet the requirements of accuracy and speed in leather surface defect detection tasks. U-Net has the advantages of small parameters and low computational complexity, but the shape and scale of leather surface defects vary greatly, and some defects are similar to background textures. Using U-Net for defect detection is prone to missed detection and false detection. Fusion and superposition of high-performance network modules and the use of feature pyramid networks can improve the detection accuracy of the model, but such methods will bring more computational costs. However, automated leather defect detection on industrial production lines faces the dilemma of limited equipment computing power. Most equipment in industrial scenarios uses embedded chips, industrial PCs, and single-board computers. Although they perform well in stability and energy efficiency, they have limited computing power and are unable to support complex image processing and deep learning models.

[0004] In view of the above problems, it is believed that it is necessary to reduce the amount of model calculation while ensuring the accuracy of model detection, so as to achieve a balance between detection accuracy and reasoning speed. 2Net integrates the advantages of the nested U-Net network and has achieved excellent results in the field of salient target detection. Based on it, this paper proposes a lightweight leather surface defect salient target detection model RBU based on the codec 2 Net. Summary of the invention

[0005] In view of the shortcomings of the existing methods, the present invention proposes a method for detecting significant targets of lightweight leather surface defects by introducing RBU.

[0006] The invention proposes a method for detecting significant targets of lightweight leather surface defects by introducing RBU, which reduces the amount of model calculation under the premise of ensuring the detection accuracy of the model, so as to achieve a balance between detection accuracy and reasoning speed. The general idea of ​​the method of the invention is to construct a lightweight feature extraction module, a neighboring feature enhancement module, a pixel hybrid sampling module, introduce a deep supervision strategy, and finally combine the modules together to construct a lightweight leather surface defect significant target detection model RBU introducing RBU 2 Net.

[0007] The lightweight feature extraction module, the residual bottleneck U-block (RBU), has a U-shaped structure that enables the module to better learn multi-scale features without reducing the resolution of the feature map, making the encoder-decoder network deeper in the embedded U-shaped network architecture.

[0008] The adjacent feature enhancement (AFE) module is used to integrate and optimize the multi-scale feature information of the adjacent stages of the encoder and decoder, enhance the model's ability to perceive the context, and avoid the direct fusion of semantically dissimilar information in simple jump connections.

[0009] The pixel hybrid sampling (PSC) module is used for up and down sampling between the stages of the codec, which improves the model's grasp of global features and increases the network's receptive field.

[0010] The deep supervision strategy solves the problem of gradient loss during the transmission process by adding additional supervision signals in the middle layers of the deep neural network, making it easier for the network to learn deep abstract features.

[0011] RBU 2 The Net model integrates rich multi-scale features and has a lightweight deep network architecture. The model uses the RBU module as the basic feature extraction unit, does not rely on any pre-trained backbone network, and is highly flexible. It can effectively learn features on data sets with fewer samples and has good adaptability to small sample data. It has significant advantages in leather surface defect detection and some similar fields of significant target detection tasks.

[0012] The present invention is specifically as follows:

[0013] In a first aspect, the present invention provides a method for detecting significant targets of lightweight leather surface defects by introducing RBU, comprising the following steps:

[0014] Step 1, obtaining a leather surface image and preprocessing it;

[0015] Step 2: Use the lightweight leather surface defect salient target detection model RBU 2 Net performs target detection on the preprocessed image;

[0016] The lightweight leather surface defect significant target detection model RBU 2 Net includes encoder, decoder, and salient image fusion module;

[0017] The encoder includes N coding layers, N>1, and adjacent coding layers are connected after pixel mixing and downsampling;

[0018] The i-th coding layer includes a first residual bottleneck U-shaped block (RBU) and a first channel addition module; the first residual bottleneck U-shaped block includes a 3×3 convolution layer, a first bottleneck block, (N+1-i) serially connected bottleneck downsampling blocks, a first channel attention block, a second bottleneck block, and (N+1-i) serially connected bottleneck upsampling blocks, i∈[1,N-1]; specifically:

[0019] The 3×3 convolution layer extracts preliminary features of the input image, and then outputs the extracted preliminary features to the channel addition module and the first bottleneck block; the first bottleneck block extracts higher-level semantic information from the received preliminary features, and then outputs it to the bottleneck upsampling block and the first bottleneck downsampling block of the same layer; the first bottleneck downsampling block reduces the size of the received image, and then outputs it to the bottleneck upsampling block and the next bottleneck downsampling block of the same layer; the second to (Ni) bottleneck downsampling blocks each receive the output feature map of the previous bottleneck downsampling block, reduce the size of the feature map, and then output the processed feature map to the bottleneck upsampling block and the next bottleneck downsampling block of the same layer; the last bottleneck downsampling block receives the output feature map of the previous bottleneck downsampling block, reduces the size of the feature map, and then outputs the processed feature map to the second bottleneck block and the first channel attention block;

[0020] The first channel attention block performs channel attention mechanism processing on the output feature map of the last bottleneck downsampling block, which helps the model learn the importance of each channel and focuses the model's attention on more critical feature channels, thereby improving the model's ability to represent input data features;

[0021] The second bottleneck block extracts more advanced semantic information based on the output feature map of the first channel attention block and the output feature map of the last bottleneck downsampling block; the first bottleneck upsampling block performs an upsampling operation based on the output feature map of the second bottleneck block and the output feature map of the bottleneck downsampling block at the same layer, and outputs it to the next bottleneck upsampling block; the second to (Ni) bottleneck upsampling blocks each receive the output feature map of the previous bottleneck upsampling block and the output feature map of the bottleneck downsampling block at the same layer, perform an upsampling operation, and output it to the next bottleneck upsampling block; the last bottleneck upsampling block receives the output feature map of the previous bottleneck upsampling block and the output feature map of the bottleneck downsampling block at the same layer, performs an upsampling operation, and outputs it to the first channel addition module;

[0022] The first channel addition module performs a channel addition operation on the output feature map of the 3×3 convolution layer and the output feature map of the last bottleneck upsampling block; the process can be expressed as:

[0023] y=F2(F1(x))+F1(x) Formula (1)

[0024] Among them, x is the input feature, y is the output feature, the F1 function is a 3×3 convolution operation, and the F2 function is the multi-scale feature extraction of RBU.

[0025] The Nth coding layer includes a second residual bottleneck U-shaped block (RBU-E) and a second channel addition module; the second residual bottleneck U-shaped block includes a 3×3 convolution layer, a third bottleneck block, a first dilated convolution block, a second dilated convolution block, a second channel attention block, a third dilated convolution block, a fourth dilated convolution block, and a fourth bottleneck block; the sizes of the first dilated convolution block and the second dilated convolution block are different, and the sizes of the third dilated convolution block and the fourth dilated convolution block are different, specifically:

[0026] The 3×3 convolution layer extracts preliminary features of the input image, and then outputs the extracted preliminary features to the second channel addition module and the third bottleneck block; the third bottleneck block extracts higher-level semantic information from the received preliminary features, and then outputs it to the first dilated convolution block and the fourth dilated convolution block; the first dilated convolution block expands the receptive field of the received output feature map, and then outputs it to the second dilated convolution block; the second dilated convolution block expands the receptive field of the received output feature map, and then outputs it to the second channel attention block; the second channel attention block processes the output feature map with a channel attention mechanism, and then outputs it to the third dilated convolution block; the third dilated convolution block expands the receptive field of the received output feature map, and then outputs it to the fourth dilated convolution block; the fourth dilated convolution block expands the receptive field of the received output feature map, and then outputs it to the fourth bottleneck block; the fourth bottleneck block extracts higher-level semantic information based on the output feature map of the second channel attention block;

[0027] The decoder includes N+1 decoding layers, and adjacent decoding layers are connected after pixel mixing upsampling; the first N decoding layers have the same structure as the encoding layers of the same layer, and the N+1 decoding layer has the same structure as the Nth decoding layer;

[0028] In the first N-1 decoding layers, the coding layer and the decoding layer in the same layer are jump-connected through the adjacent feature enhancement module (AFE); the N+1 decoding layer receives the output feature map of the N coding layer, decodes it, and then sends the output to the N decoding layer;

[0029] The adjacent feature enhancement module (AFE) concatenates the output feature map of the current coding layer after pixel mixing and downsampling with the output feature map of the next coding layer in series at the channel level, and then fuses and enhances the feature information of the two stages through a convolution layer; the calculation formula of this process is as follows:

[0030] M i =Conv(Concat(E i ,E i+1 )) Formula (2)

[0031] Feature map M enhanced by adjacent feature enhancement module (AFE) i The feature map after pixel mixing and upsampling of the next layer decoder is connected in series and used as the input of the current layer decoder; the calculation formula of this process is as follows:

[0032] D i =Concat(M i ,PSC up (D i+1 )) Formula (3)

[0033] Among them, E i and D i denote the encoder and decoder of the i-th layer respectively, M i It represents the feature map calculated by the AFE module, Concat represents the concatenation of feature maps in the channel dimension, Conv represents the convolution operation, and PSC up Indicates pixel blending upsampling.

[0034] The salient image fusion module performs pixel-mixing upsampling on the salient target probability map generated by each layer of the decoder, restores it to the same resolution as the input image, and then uses a 3×3 convolution layer and a sigmoid activation function to map the feature maps of all stages into salient target images; then the salient target images output by different stages of the decoder are concatenated at the channel level, and then a 1×1 convolution layer and an activation function are used to obtain the final salient target prediction map.

[0035] In a second aspect, the present invention provides a leather surface defect significant target detection system, comprising:

[0036] The data acquisition module is responsible for acquiring the leather surface image and preprocessing it;

[0037] The target detection module is responsible for using the lightweight leather surface defect salient target detection model RBU 2 Net performs target detection on the preprocessed image.

[0038] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described.

[0039] In a fourth aspect, the present invention provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the described method is implemented.

[0040] Beneficial effects of the present invention:

[0041] The present invention proposes a lightweight leather surface defect significant target detection method that introduces RBU, and realizes a significant target detection method for leather surface defects. In order to realize the defect detection of lightweight model and real-time performance, a residual bottleneck U-shaped feature extraction unit is designed, which not only combines the structural advantages of U-shaped network and bottleneck block, but also integrates the performance advantages of deep separable convolution and channel attention. While ensuring the model representation ability, the model parameters are significantly reduced, and the reasoning speed of the model on the development board is improved, which solves the dilemma of limited computing equipment in the automatic leather defect detection on the industrial production line to a certain extent. For the adjacent feature extraction units in the codec, a neighboring feature enhancement module is designed to enhance the model's ability to fuse multi-scale features and optimize the current situation in the field of leather detection that cannot accurately characterize the defect shape. For the problem of information loss in the up and down sampling process, a pixel hybrid sampling module is designed to improve the utilization rate of feature information and enhance the spatial perception ability of the model, so that it can improve the accuracy of leather detection when detecting surface defects of leather with various types, irregular defect shapes, and a large range of scale changes. In addition, the strategy of deep supervision and mixed loss function is adopted, combined with effective data enhancement technology, to further enhance the learning ability of the model.

[0042] The present invention meets the real-time and fine-grained requirements of the target task, and is of great significance for the automated leather defect detection scenario on industrial production lines with limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 RBU of the present invention2 Net network architecture diagram;

[0044] Figure 2 This is a structural diagram of the feature extraction module of the present invention;

[0045] Figure 3 The adjacent feature enhancement module structure of the present invention;

[0046] Figure 4 This is a downsampling flow chart of the pixel hybrid sampling module of the present invention;

[0047] Figure 5 This is a flow chart of upsampling of the pixel hybrid sampling module of the present invention;

[0048] Figure 6 This is a schematic diagram of data enhancement of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be described below. Figure 1 RBU 2 Net network architecture diagram for specific explanation.

[0050] The present invention provides a method for detecting significant targets of lightweight leather surface defects by introducing RBU, comprising the following steps:

[0051] Step 1: Resize each batch of original leather surface images and GT (Ground Truth) images to a uniform resolution of 640×640 to ensure the standardization of the input. Then, crop the images to 576×576 tensors and input them into the model for training.

[0052] According to the model's demand for sample quantity, in order to make full use of limited data resources, data enhancement techniques such as Figure 6 ,By performing a series of transformations and expansions on the original data, additional training samples are generated, thereby enriching the diversity of the data.

[0053] Step 2: Use the lightweight leather surface defect salient target detection model RBU 2 Net performs target detection on the preprocessed image;

[0054] The lightweight leather surface defect significant target detection model RBU 2 Net includes encoder, decoder, and salient image fusion module;

[0055] The encoder contains 5 encoding layers, which extract features of the input image in stages from top to bottom. The basic structure of the encoder is as follows: Figure 1The En-1, En-2, En-3, and En-4 layers shown include the first residual bottleneck U-shaped block RBU-1+first channel addition module, the first residual bottleneck U-shaped block RBU-2+first channel addition module, the first residual bottleneck U-shaped block RBU-3+first channel addition module, and the first residual bottleneck U-shaped block RBU-4+first channel addition module, respectively. The first residual bottleneck U-shaped block RBU is used as a feature extraction module. RBU-1 includes a 3×3 convolution layer, a first bottleneck block, 5 serial bottleneck downsampling blocks, a channel attention block, a second bottleneck block, and 5 serial bottleneck upsampling blocks; RBU-2 includes a 3×3 convolution layer, a first bottleneck block, and 5 serial bottleneck downsampling blocks. The RBU-3 includes a 3×3 convolution layer, the first bottleneck block, 3 serial bottleneck downsampling blocks, a channel attention block, the second bottleneck block, and 3 serial bottleneck upsampling blocks; the RBU-4 includes a 3×3 convolution layer, the first bottleneck block, 2 serial bottleneck downsampling blocks, a channel attention block, the second bottleneck block, and 2 serial bottleneck upsampling blocks; the configuration of the number L of bottleneck upsampling blocks and bottleneck downsampling blocks is related to the resolution of the input features. For input feature maps with higher resolution, more layers are used for deeper feature extraction. In the encoder En-5 stage, En-5 includes a second residual bottleneck U-shaped block and a second channel addition module; the second residual bottleneck U-shaped block includes a 3×3 convolution layer, a third bottleneck block, a first dilated convolution block, a second dilated convolution block, a second channel attention block, a third dilated convolution block, a fourth dilated convolution block, and a fourth bottleneck block; the sizes of the first dilated convolution block and the second dilated convolution block are different, and the sizes of the third dilated convolution block and the fourth dilated convolution block are different, wherein the dilation of the first dilated convolution block and the third dilated convolution block is 2, and the dilation of the second dilated convolution block and the fourth dilated convolution block is 4, the value of dilation represents the number of null values ​​added between each element of the convolution kernel, and dilated convolution is used to replace the bottleneck up and down sampling blocks in the En-1 to En-4 stages, and all intermediate feature layers have the same resolution as the input and output feature layers.

[0056] Specifically, the U-shaped structure of the first residual bottleneck U-shaped block (RBU) enables the module to better learn multi-scale features without reducing the resolution of the feature map, so that the codec network can be deeper in the embedded U-shaped network architecture; the first residual bottleneck U-shaped block is a 3×3 convolution layer that extracts preliminary features of the input image, and then outputs the extracted preliminary features to the channel addition module and the first bottleneck block; the first bottleneck block extracts higher-level semantic information from the received preliminary features, and then outputs them to the bottleneck upsampling block and the first bottleneck downsampling block of the same layer; the second to (Ni) bottleneck downsampling blocks each receive the output feature map of the previous bottleneck downsampling block, reduce the size of the feature map, and then output the processed feature map to the bottleneck upsampling block and the next bottleneck downsampling block of the same layer; the last bottleneck downsampling block receives the output feature map of the previous bottleneck downsampling block, The feature map size is reduced, and then the processed feature map is output to the second bottleneck block and the channel attention block; the second bottleneck block extracts more advanced semantic information according to the output feature map of the channel attention block and the output feature map of the last bottleneck downsampling block; the first bottleneck upsampling block performs an upsampling operation according to the output feature map of the second bottleneck block and the output feature map of the bottleneck downsampling block of the same layer, and outputs it to the next bottleneck upsampling block; the second to (Ni) bottleneck upsampling blocks each receive the output feature map of the previous bottleneck upsampling block and the output feature map of the bottleneck downsampling block of the same layer for upsampling operation, and output it to the next bottleneck upsampling block; the last bottleneck upsampling block receives the output feature map of the previous bottleneck upsampling block and the output feature map of the bottleneck downsampling block of the same layer for upsampling operation, and outputs it to the first channel addition module; the upsampling operation is a bilinear interpolation method;

[0057] The second residual bottleneck U-block is specifically:

[0058] The 3×3 convolution layer extracts preliminary features of the input image, and then outputs the extracted preliminary features to the second channel addition module and the third bottleneck block; the third bottleneck block extracts higher-level semantic information from the received preliminary features, and then outputs it to the first dilated convolution block and the fourth dilated convolution block; the first dilated convolution block expands the receptive field of the received feature map, and then outputs it to the second dilated convolution block; the second dilated convolution block expands the receptive field of the received feature map, and then outputs it to the second channel attention block; the second channel attention block processes the feature map with a channel attention mechanism, and then outputs it to the third dilated convolution block; the third dilated convolution block expands the receptive field of the received feature map, and then outputs it to the fourth dilated convolution block; the fourth dilated convolution block expands the receptive field of the received feature map, and then outputs it to the fourth bottleneck block; the fourth bottleneck block extracts higher-level semantic information based on the output feature map of the second channel attention block;

[0059] like Figure 2 The first bottleneck block, the second bottleneck block, the third bottleneck block, the fourth bottleneck block, and the bottleneck upsampling block have the same structure, all using Bottleneck_1; the bottleneck downsampling block uses Bottleneck_2. In Bottleneck_1, the stride is set to 1, and the feature map resolution size is not changed. In Bottleneck_2, the pooling operation is abandoned, and the image resolution is reduced by using a depthwise separable convolution with a stride of 2; in this way, the model receptive field is increased, and the extra computing time required for the pooling operation is saved. The implementation of Bottleneck_1 first compresses the number of feature channels through a 1×1 convolution layer to reduce the amount of model calculation and increase the network training and inference speed; followed by a 3×3 depthwise separable convolution, and then a 1×1 convolution layer is used to expand the number of channels, so that the model has more opportunities to retain valid information. In addition, residual connections are used to avoid gradient disappearance or gradient explosion. The implementation of Bottleneck_2 first compresses the number of feature channels through a 1×1 convolution layer, followed by a 3×3 depthwise separable convolution, and then uses a 1×1 convolution layer to expand the number of channels; in addition, its residual connection uses the result of the 3×3 depthwise separable convolution of the input of the bottleneck downsampling block as the output.

[0060] The first and second channel attention blocks perform channel attention mechanism processing on the output feature map of the last bottleneck downsampling block, which helps the model learn the importance of each channel and focuses the model's attention on more critical feature channels, thereby improving the model's ability to represent input data features; the channel attention mechanism helps the model learn the importance of each channel and focuses the model's attention on more critical feature channels, thereby improving the model's ability to represent input data features; the channel attention mechanism is used in the bottom layer of the RBU module. Since the feature map scale of the bottom layer is very small after multiple downsamplings, the use of the channel attention mechanism at this time does not introduce too much time cost, and while enhancing the model's feature extraction capability, it also controls the model parameter quantity and computational complexity;

[0061] The first and second channel addition modules perform channel addition operations on the output of the 3×3 convolution layer and the output of the last bottleneck upsampling block as the output of the RBU. This calculation process can be expressed as:

[0062] y=F2(F1(x))+F1(x) Formula (1)

[0063] Among them, x is the input feature, y is the output feature, the F1 function is a 3×3 convolution operation, and the F2 function is the multi-scale feature extraction of RBU.

[0064] After the encoder extracts features from the input image in stages from top to bottom, the bottom layer of the outer U-shaped network is the De-6 stage of the decoder, and the decoder fuses the features of different stages from bottom to top. De-6 in the decoder and each stage of the subsequent decoder participate in deep supervision and the output of saliency maps at the same time, so this deep feature extraction module is divided into the decoder. De-6 and De-5 use the same RBU+channel addition module as En-5, and do not perform feature downsampling. The De-4, De-3, De-2 and De-1 stages of the decoder use the same RBU+channel addition module as the encoding layer of the same stage, and the same RBU is RBU-4, RBU-3, RBU-2 and RBU-1. In the decoder, the output features of the decoder in the previous stage are upsampled by pixel mixing, and are connected in series with the features output by the AFE module of the corresponding encoder stage as the input features of the decoder in the next stage.

[0065] like Figure 3 , a neighboring feature enhancement module is designed between the jump connections of the same layer of the encoder and decoder, so that the model can better fuse the multi-scale features of different stages and further optimize the model performance. The neighboring feature enhancement (AFE) module integrates and optimizes the multi-scale feature information of the adjacent stages of the encoder and decoder, enhances the model's ability to perceive the context, and avoids the direct fusion of semantically dissimilar information in simple jump connections; the neighboring feature enhancement module (AFE) concatenates the output feature map of the current coding layer after pixel mixing and downsampling operations with the output feature map of the next coding layer at the channel level, and then fuses and enhances the feature information of the two stages through a convolutional layer. The calculation formula for this process is as follows:

[0066] M i =Conv(Concat(E i ,E i+1 )) Formula (2)

[0067] Feature map M enhanced by adjacent feature enhancement module (AFE) i The feature map after pixel mixing and upsampling of the next layer decoder is connected in series and used as the input of the current layer decoder; the calculation formula of this process is as follows:

[0068] D i =Concat(M i ,PSC up (D i+1 )) Formula (3)

[0069] Among them, E i and D i denote the encoder and decoder of the i-th layer respectively, M iIt represents the feature map calculated by the AFE module, Concat represents the concatenation of feature maps in the channel dimension, Conv represents the convolution operation, and PSC up Indicates pixel blending upsampling.

[0070] RBU 2 In Net, pixel mixing upsampling and pixel mixing downsampling are used instead of the maximum pooling layer and bilinear interpolation method for ups and downsampling between stages, which can make more full use of feature information. Pixel mixing upsampling and pixel mixing downsampling can be considered as pixel mixing sampling (PSC), which is used for ups and downsampling between the stages of the codec, improving the model's grasp of global features and increasing the network's receptive field;

[0071] like Figure 4 , the process of pixel hybrid downsampling is as follows: the high-resolution feature map with a height of H and a width of W is divided into 2×2 windows according to pixels, and the pixels at the same position of the window are extracted into the feature map of the same channel, and the pixels at different positions of the window are extracted into different channels; these pixels are rearranged according to the relative positions of the windows, and four low-resolution feature maps with a height of H / 2 and a width of W / 2 are obtained; in this way, the number of feature channels is expanded to 4 times the original number, and the number of channels will increase sharply when multiple PSC modules are superimposed; therefore, a depth-separable convolution layer is added after Pixel Shuffle, which not only controls the number of channels but also increases the model receptive field without significantly increasing the amount of module calculation; let the input high-resolution feature be X, then the calculation formula of pixel hybrid downsampling is as follows:

[0072]

[0073] Among them, DWC onv Function represents depth-wise separable convolution, PS down Indicates Pixel Shuffle downsampling. C in and C out Respectively represent the number of input and output channels;

[0074] Upsampling is required between each stage of the decoder to perform multi-scale feature fusion; in the U-shaped network architecture, upsampling and downsampling play an equally important role;

[0075] like Figure 5 , pixel hybrid upsampling process, this module uses 4 n×n low-resolution feature maps to map into a 2n×2n high-resolution feature map through the PixelShuffle operation; then through a depth-separable convolution layer, the number of output channels is controlled while the receptive field is expanded; let the input high-resolution feature be X, then the calculation formula for pixel hybrid upsampling is as follows:

[0076]

[0077] Among them, the DWConv function represents the depth-separable convolution, PS up Indicates Pixel Shuffle upsampling; C in and C out Respectively represent the number of input and output channels.

[0078] Each stage of the decoder will output a salient image for deep supervision of the entire network. The salient images output by each stage of the decoder can be combined to obtain the final salient object detection result.

[0079] The residual bottleneck U-shaped feature extraction module is embedded in the outer U-shaped network framework to obtain RBU 2 Net’s basic architecture, the nested U-shaped structure can realize multi-scale feature extraction within a stage and multi-scale feature aggregation between stages. A neighboring feature enhancement module is designed between the jump connections of the same layer of the encoder and decoder, so that the model can better integrate multi-scale features of different stages and further optimize the model performance. 2 In Net, pixel hybrid sampling is used instead of the maximum pooling layer and bilinear interpolation method for up and down sampling between stages, which can make more full use of feature information. During model training, the use of deep supervision strategy and hybrid loss function effectively avoids gradient vanishing and gradient explosion, improves the network representation ability and speeds up the model convergence.

[0080] The model ends with the salient object image fusion module. In each training or inference, each stage of the decoder generates a salient object probability map. 2 Net restores the output features of each decoder stage to the same resolution as the input image by pixel mixing upsampling, and then uses a 3×3 convolutional layer and a sigmoid activation function to map the feature maps of these six stages into a salient target image (see Figure 1 Medium S (1) stage , S (2) stage , S (3) stage , S (4) stage , S (5) stage and S (6) stage ). The salient target images output by the six stages of the decoder can also play a role in deep supervision during training. Finally, the salient maps output by different stages of the decoder are connected, and then a 1×1 convolution layer and activation function are used to obtain the final salient target prediction map (see Figure 1 Medium S fuse ).

[0081] Considering that the depth of neural network can improve the network representation ability to a certain extent, but as the depth continues to increase, training becomes more difficult and the risk of gradient disappearance and gradient explosion also increases, a deep supervision strategy is introduced; Lightweight leather surface defect significant target detection model RBU 2 Net uses a deep supervision loss function It is expressed as follows:

[0082]

[0083] in, It is the output saliency map at different stages of the decoder The total loss, It is the saliency map S output after the final fusion of feature maps at each stage fuse The total loss, and ω fuse They are the loss weight of each stage and the loss weight of the final result;

[0084] The total loss l at each stage is calculated using a mixed loss function of cross entropy loss and intersection-over-union loss total Combining these two loss functions can achieve better monitoring of global semantics and local details, and improve the performance and stability of the model; the total loss function of each layer output saliency map can be expressed as:

[0085]

[0086] Among them, μ is a hyperparameter that adjusts the weight of IOU loss in the total loss, and n represents the nth layer.

[0087] The total loss function of the stage The binary cross entropy loss formula in is as follows:

[0088]

[0089] Where H and W are the height and width of the saliency map output at different stages of the decoder, respectively, and G ij Represents the pixel value of the true label, P ij Represents the pixel values ​​of the saliency prediction map output at different stages of the decoder;

[0090] The total loss function of the stage The loss formula of intersection-over-union ratio is as follows:

[0091]

[0092] Where H and W are the height and width of the saliency map output at different stages of the decoder, respectively, and G ijRepresents the pixel value of the true label, P ij Represents the pixel values ​​of the saliency prediction map output at different stages of the decoder;

[0093] Finally, the saliency map S is output after the feature maps of each stage are fused. fuse The total loss function can be expressed as:

[0094]

[0095] Among them, μ′ is a hyperparameter that adjusts the weight of IOU loss in the total loss.

[0096] The saliency map S is output after the final fusion of the feature maps of each stage. fuse The total loss function The binary cross entropy loss formula in is as follows:

[0097]

[0098] Among them, H′ and W′ are the height and width of the saliency map image output after the final fusion of the feature maps of each stage, G ij ′ represents the pixel value of the true label, P ij ′ represents the pixel value of the significant prediction map output after the feature maps of each stage are finally fused;

[0099] The saliency map S is output after the final fusion of the feature maps of each stage. fuse The total loss function The loss formula of intersection-over-union ratio is as follows:

[0100]

[0101] Among them, H′ and W′ are the height and width of the saliency map image output after the final fusion of the feature maps of each stage, G ij ′ represents the pixel value of the true label, P ij ′ represents the pixel value of the salient prediction map output after the feature maps of each stage are finally fused.

[0102] Step 3: RBU 2 Net model for comparative experiments.

[0103] Model RBU 2 Net is compared with the mainstream methods in the field of salient object detection in recent years, including UCF, AFNet, SCRN, SRM, BASNet, U 2 Net and U 2Net-s. The dataset LDSOD Dataset used comes from the leather material production line of Gujia Home Furnishing. After collection and processing, a total of 2136 high-definition leather defect images were obtained, including 1709 training sets and 427 test sets. Leather defects are divided into seven categories: holes, scratches, acne spots, creases, indentations, rotten surfaces, and abnormal patterns. Each image is manually annotated with high quality.

[0104] Table 1 Quantitative experimental results of different salient object detection methods on LDSOD Dataset

[0105]

[0106]

[0107] Table 1 shows the quantitative experimental results of different salient object detection methods on the LDSOD Dataset. ↑ indicates that the larger the index, the better the performance, ↓ indicates that the smaller the index, the better the performance, the bold data indicates the best performance, and the underlined data indicates the second best performance. 2 Net-s performs best in detection speed. 2 Net-s is U 2 Net, but due to its small size, it lacks the ability to represent the features of leather defect data and shows disadvantages in detection accuracy. BASNet emphasizes the ability to perceive the boundary of the target. m The best results were achieved in terms of indicators, but due to the complex network structure, it brings more computing costs, so it is not suitable for industrial application scenarios with high speed requirements. 2 Although Net did not take the lead in performance and speed, it ranked second in F1 and MAE indicators, and has achieved a good balance between speed and accuracy, but has not yet reached the requirements of practical applications. 2 Net is based on U 2 Net architecture, while ensuring the accuracy of the model, pays more attention to the model's reasoning speed so that it can be applied to industrial scenarios. Experiments show that RBU 2 Net has a strong advantage in speed, compared with U 2 Net increased by 8.2 frames per second (FPS), achieved the best results in F1 and MAE indicators, and m Compared with the fastest U 2 Net-s, RBU 2 Net in F1 and S m The indicators are leading by 7.1% and 4.4% respectively, and MAE is reduced by 28.1%, achieving a balance between inference speed and detection accuracy.

Claims

1. A method for detecting significant defects on lightweight leather surface by introducing RBU, the method comprising the following steps: Step 1, obtaining a leather surface image and preprocessing it; Step 2: Use the lightweight leather surface defect salient target detection model RBU 2 Net performs target detection on the preprocessed image; It is characterized in that The lightweight leather surface defect significant target detection model RBU 2 Net includes encoder, decoder, and salient image fusion module; The encoder includes N coding layers, and adjacent coding layers are connected after pixel mixing and downsampling; the i-th coding layer includes a first residual bottleneck U-shaped block and a first channel addition module, i∈[1,N-1]; the first residual bottleneck U-shaped block includes a 3×3 convolution layer, a first bottleneck block, (N+1-i) serially connected bottleneck downsampling blocks, a first channel attention block, a second bottleneck block, and (N+1-i) serially connected bottleneck upsampling blocks; the N-th coding layer includes a second residual bottleneck U-shaped block and a second channel addition module; the second residual bottleneck U-shaped block includes a 3×3 convolution layer, a third bottleneck block, a first dilated convolution block, a second dilated convolution block, a second channel attention block, a third dilated convolution block, a fourth dilated convolution block, and a fourth bottleneck block; the first dilated convolution block and the second dilated convolution block have different sizes, and the third dilated convolution block and the fourth dilated convolution block have different sizes; The decoder includes N+1 decoding layers, and adjacent decoding layers are connected after pixel mixing upsampling; the first N decoding layers have the same structure as the encoding layers of the same layer, and the N+1th decoding layer has the same structure as the Nth decoding layer.

2. The method according to claim 1, characterized in that The specific implementation process of the first residual bottleneck U-shaped block is: the 3×3 convolution layer extracts preliminary features of the input image, and then outputs the extracted preliminary features to the first channel addition module and the first bottleneck block; The first bottleneck block extracts higher-level semantic information from the received preliminary features and then outputs them to the bottleneck upsampling block and the first bottleneck downsampling block in the same layer; The first bottleneck downsampling block reduces the size of the received feature map and then outputs it to the bottleneck upsampling block and the next bottleneck downsampling block in the same layer; the second to (Ni) bottleneck downsampling blocks each receive the output feature map of the previous bottleneck downsampling block, reduce the size of the feature map, and then output the processed feature map to the bottleneck upsampling block and the next bottleneck downsampling block in the same layer; the last bottleneck downsampling block receives the output feature map of the previous bottleneck downsampling block, reduces the size of the feature map, and then outputs the processed feature map to the second bottleneck block and the first channel attention block; The first channel attention block performs a channel attention mechanism on the output feature map of the last bottleneck downsampling block; The second bottleneck block extracts more advanced semantic information based on the output feature map of the first channel attention block and the output feature map of the last bottleneck downsampling block; The first bottleneck upsampling block performs an upsampling operation according to the output feature map of the second bottleneck block and the output feature map of the bottleneck downsampling block at the same layer, and outputs it to the next bottleneck upsampling block; the second to (Ni) bottleneck upsampling blocks each receive the output feature map of the previous bottleneck upsampling block and the output feature map of the bottleneck downsampling block at the same layer, perform an upsampling operation, and output it to the next bottleneck upsampling block; the last bottleneck upsampling block receives the output feature map of the previous bottleneck upsampling block and the output feature map of the bottleneck downsampling block at the same layer, performs an upsampling operation, and outputs it to the first channel addition module; The first channel addition module performs a channel addition operation on the output feature map of the 3×3 convolutional layer and the output feature map of the last bottleneck upsampling block.

3. The method according to claim 1, characterized in that: In the first N-1 decoding layers, the encoding layer and the decoding layer at the same layer are jump-connected through adjacent feature enhancement modules; the N+1 decoding layer receives the output feature map of the N encoding layer, decodes it, and then sends the output to the N decoding layer.

4. The method according to claim 3, characterized in that: The implementation process of the adjacent feature enhancement module is: the result of the pixel mixing downsampling operation on the output feature map of the current coding layer is concatenated with the output feature map of the next coding layer at the channel level, and then the feature information of the two stages is fused and enhanced through a convolution layer; the calculation formula of this process is as follows: M i =Conv(Concat(E i ,E i+1 )) Formula (1).

5. The method according to claim 3, characterized in that: The input of each layer of the decoder is the feature map M enhanced by the adjacent feature enhancement module i The result is concatenated with the feature map after pixel mixing and upsampling of the next layer of decoder; the calculation formula for this process is as follows: D i = Concat(M i , PSC up (D i+1 )) Equation (2) Among them, E i and D i denote the encoder and decoder of the i-th layer respectively, M i It represents the feature map calculated by the AFE module, Concat represents the concatenation of feature maps in the channel dimension, Conv represents the convolution operation, and PSC up Indicates pixel blending upsampling.

6. The method according to claim 3, characterized in that: The salient image fusion module performs pixel-mixing upsampling on the salient target probability map generated by each layer of the decoder, restores it to the same resolution as the input image, and then uses a 3×3 convolution layer and a sigmoid activation function to map the feature maps of all stages into salient target images; then the salient target images output by different stages of the decoder are concatenated at the channel level, and then a 1×1 convolution layer and an activation function are used to obtain the final salient target prediction map.

7. The method according to claim 1, characterized in that Lightweight leather surface defect salient target detection model RBU 2 Net uses a deep supervision loss function It is expressed as follows: in, It is the output saliency map at different stages of the decoder The total loss, It is the total saliency map S output after fusing the feature maps of each stage fuse The total loss, and ω fuse They are and The loss weight of Among them, μ is a hyperparameter, n represents the nth layer decoder, Represents the output saliency map at different stages The binary cross entropy loss is Represents the output saliency map at different stages The intersection-over-union loss is denoted by ∑ i = 1, i = 1; H and W are the height and width of the saliency map image output at different stages of the decoder, respectively; G ij Represents the pixel value of the true label of the saliency map output at different stages, P ij Represents the pixel values ​​of the saliency prediction map output at different stages of the decoder; Among them, μ′ is a hyperparameter, Represents the total saliency map S fuse Binary cross entropy loss, Represents the total saliency map S fuse Intersection-over-Union loss; H′ and W′ are the total saliency map S fuse The height and width, G ij ′ represents the total saliency map S fuse The pixel value of the true label, P ij ′ represents the total saliency map S fuse The pixel value of .

8. A leather surface defect salient target detection system implementing the method according to any one of claims 1 to 7, characterized in that include: The data acquisition module is responsible for acquiring the leather surface image and preprocessing it; The target detection module is responsible for using the lightweight leather surface defect salient target detection model RBU 2 Net performs target detection on the preprocessed image.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

10. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 7 is implemented.

Images