Label text detection method based on multi-level separation feature enhancement and spatial attention

Abstract

The present invention discloses a label text detection method based on multi-level separation feature enhancement and spatial attention. Obtain an industrial product label text image and preprocess it, put it into ResNet50HS‑Block, obtain a sufficient receptive field, and obtain a multi-scale feature map; adopt a dual-branch design concept, send the feature map to SAFFMG and CFPEM respectively to extract multi-scale feature information, and then perform simple splicing and convolution operations on the feature map generated by the fusion dual branches, and predict the instance vector, text area, and text kernel through the detection head Head; obtain the detection result of the instance vector, text area, and text kernel through the pixel aggregation algorithm PA of the fusion background. The method of the present invention is applicable to various complex factory environments, as well as unrestricted industrial labels with different shapes and positions. The accuracy of the adopted network model is significantly higher than that of the existing model when the detection speed meets the basic requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a label text detection method based on multi-level separation feature enhancement and spatial attention. Background Art

[0002] Text detection refers to marking the area where the text is located in a natural scene image containing text, such as detecting the text area in a billboard. In addition, text detection has important practical application value in computer vision tasks such as scene understanding, autonomous driving, and human-computer interaction, which has attracted widespread attention and research from researchers. However, the complex image background and the changing text form greatly increase the challenge of this task. In the process of transmitting industrial products on the mobile production line, the factory packs the products and affixes industrial labels, and then transports the boxes to the shipping area. Although workers record product information, it is still inevitable that the label information does not match the actual required information due to worker negligence. Once this happens, it will lead to a series of product shipping errors, which will cost a considerable amount of cost to investigate later, seriously affecting cargo management and production efficiency.

[0003] In order to improve the accuracy of shipped products, text detection technology is used to accurately detect the text in product labels. However, applying text detection technology to this problem scenario requires overcoming the following difficulties: (1) The factory environment is complex with dim lighting, obstructions, etc. Industrial labels have the characteristics of large amount of text, different text languages, and dense text. In addition, due to the different shapes of industrial products, the sizes of cargo boxes are different, and the locations of industrial labels are also different. Therefore, the network needs to ensure a high accuracy rate. (2) The industrial production line is in motion, but the position and angle of the camera are fixed. Therefore, in order not to affect the normal operation of the assembly line, while ensuring a high accuracy rate, it is also necessary to complete the detection and other tasks within a certain time. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a label text detection method based on multi-level separation feature enhancement and spatial attention, which achieves considerable accuracy while meeting the basic requirements of detection speed.

[0005] In a first aspect, the present invention provides a label text detection method based on multi-level separation feature enhancement and spatial attention, comprising the following steps:

[0006] Step (1), photographing the industrial product, obtaining the industrial product label text image, and preprocessing the industrial product label text image;

[0007] Step (2), putting the preprocessed text image into a multi-level separation enhanced ResNet model (ResNet50HS-Block) to obtain a sufficient receptive field and obtain a multi-scale feature map;

[0008] The ResNet50 HS-Block uses the ResNet50 network as the skeleton and replaces the original 3×3 convolution module with the multi-level separation module Hierarchical-Split Block;

[0009] The Hierarchical-Split Block divides the received feature map into 4 groups in the channel dimension through the Split operation. Each group of feature maps X i The first group retains the original input feature map X1; the second group first performs convolution operation Conv on the input feature map X2, batch normalization operation, and increases network nonlinearity through Relu activation function to obtain Y2, and then performs Split operation on Y2 to divide it into two parts Y 2,1 and Y 2,2 , output Y 2,1 ; Group 3 will be Y 2,2 Concat with the input feature map X3, then perform convolution operation Conv on the concatenated feature map, batch normalization operation, and increase network nonlinearity through Relu activation function to obtain Y3, and then split Y3 into two parts, output Y 3,1 , and Y 3,2 Concat with the next set of input feature maps; the fourth group will be Y 3,2 Concat with the input feature map X4, then perform convolution operation Conv on the concatenated feature map, perform batch normalization operation, and increase network nonlinearity through Relu activation function; finally, concat the outputs of the four groups to obtain a multi-scale feature map;

[0010] Y i The calculation formula is as follows:

[0011]

[0012] Since the first group does not perform the Split operation, Y 1,2 =0;

[0013] Step (3), using the dual-branch design idea, the multi-scale feature map output by ResNet50 HS-Block is sent to the spatial attention-based feature fusion module SAFFMG and the progressive fusion feature pyramid module CFPEM to extract multi-scale feature information;

[0014] The spatial attention-based feature fusion module SAFFMG is used to extract multi-scale feature information, specifically:

[0015] The feature maps of different scales output by the ResNet50 HS-Block are scaled to the same resolution after upsampling and stacking operations to obtain feature maps Then the above feature map X 1 Concatenate and then perform a 3×3 convolution to obtain the intermediate features Input the intermediate feature M into the spatial attention module and calculate the spatial attention weight According to the spatial attention weight For N feature maps Weighted, get fusion features

[0016] The spatial attention module performs maximum pooling and average pooling operations on the input features, then performs concat splicing, and then fuses the spliced ​​features with the original input features again, and then performs a convolution to obtain the spatial attention weights.

[0017] The progressively fused feature pyramid module CFPEM includes a feature pyramid enhancement module FPEM and a feature fusion module FFM; the feature pyramid enhancement module FPEM consists of two stages, namely upscaling enhancement and downscaling enhancement; in the output stage of upscaling enhancement, the input of the current module and the features generated by downscaling enhancement are added element by element to obtain the result; the feature fusion module FFM performs feature fusion on all the results of the feature pyramid enhancement module FPEM;

[0018] Step (4) After simple concatenation and convolution operations on the feature maps output by the spatial attention-based feature fusion module SAFFMG and the progressive fusion feature pyramid module CFPEM, the corresponding instance vector, text area, and text kernel are obtained by pre-detection by a lightweight detection head Head; the instance vector, text area, and text kernel are obtained by a pixel aggregation algorithm PA that fuses the background to obtain the final detection result; specifically:

[0019] Different instance vectors are regarded as different clusters, the text core is the cluster center, and the pixels in the text area are the samples to be clustered; in the same text line, in order to guide the pixels to the corresponding text core, the distance between the two should be minimized;

[0020] During the training phase, the formula for the clustering loss function is:

[0021]

[0022]

[0023]

[0024] Where N is the number of text lines, R i represents the i-th text region, K i Represents a text line R i The corresponding kernel, Represents text pixel r and text kernel K i The distance between them, the Relu function is used to ensure that the calculation is non-negative, represents the instance vector corresponding to pixel r, Represents the text kernel K i The corresponding instance vector, λ clu is a constant with a value of 0.5;

[0025] The cluster center needs to ensure sufficient discrimination. Therefore, the instance vector of the text core needs to maintain a certain distance from the adjacent text core. On this basis, the distinction of background pixels is introduced, and the recognition loss function is as follows:

[0026]

[0027]

[0028]

[0029] Among them, P bg represents the instance vector of the background pixel, represents the distance between the text kernel and the background vector, Represents the text kernel K i With K j The distance between dis is a constant with a value of 3.

[0030] In a second aspect, the present invention provides a label text detection system based on multi-level separation feature enhancement and spatial attention, comprising:

[0031] An image acquisition and preprocessing module, used to acquire industrial product label text images and preprocess the industrial product label text images;

[0032] ResNet50 HS-Block module, used to extract multi-scale feature maps from preprocessed text images;

[0033] The spatial attention-based feature fusion module SAFFMG is used to extract multi-scale feature information F;

[0034] The progressive fusion feature pyramid module CFPEM is used to extract multi-scale fusion feature information;

[0035] The fusion module is used to concatenate and convolve the feature maps output by the spatial attention-based feature fusion module SAFFMG and the progressive fusion feature pyramid module CFPEM to obtain the final features;

[0036] The lightweight detection head obtains the corresponding instance vector, text area, and text kernel based on the final feature measurement;

[0037] The output module is used to obtain the final detection result by integrating the instance vector, text area, and text kernel through the pixel aggregation algorithm PA with the background.

[0038] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described.

[0039] In a fourth aspect, the present invention provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the described method is implemented.

[0040] The beneficial effects of the present invention are:

[0041] The present invention proposes a dual-branch concept and introduces a spatial attention-based feature fusion module SAFFMG and a progressive fusion feature pyramid module CFPEM to dynamically fuse features of different scales to adapt to the complexity of the factory environment and make full use of the spatial and global information of features of different levels and scales in the process of decoding feature fusion.

[0042] When extracting multi-scale features, the present invention uses a multi-level separation module Hierarchical-Split Block to improve the traditional 3×3 convolution in the residual block of the ResNet50 network, thereby improving the accuracy while maintaining a considerable amount of calculation.

[0043] The method of the present invention is applicable to various complex factory environments and is not limited by the shapes and positions of industrial tags. The accuracy of the adopted network model is significantly higher than that of the existing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the method of the present invention;

[0045] Figure 2 This is the structural diagram of the ResNet50 HS-Block of the present invention;

[0046] Figure 3It is a structural diagram of the feature fusion module SAFFMG based on spatial attention of the present invention;

[0047] Figure 4 It is a structural diagram of the progressively fused feature pyramid module CFPEM of the present invention;

[0048] Figure 5 It is a structural diagram of the lightweight detection head module of the present invention;

[0049] Figure 6 It is a structural diagram of the pixel aggregation module based on kernel representation of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings.

[0051] Figure 1 An industrial label text detection model based on multi-level separation feature enhancement and attention mechanism is given. In order to ensure sufficient accuracy, the model uses ResNet50 as the backbone network, and uses the multi-level separation module Hierarchical-Split Block to improve the conventional convolution module, combining the advantages of Ghost network and Res2Net network to obtain a more adequate receptive field. Secondly, the dual-branch design idea is adopted to send the feature map to the spatial attention-based feature fusion module SAFFMG and the progressive fusion feature pyramid module CFPEM to extract multi-scale feature information, and then the feature map generated by the fusion dual branches is simply spliced ​​and convolved to generate a lightweight detection head Head, which predicts instance vectors, text areas, and text kernels; the instance vectors, text areas, and text kernels are obtained by the pixel aggregation algorithm PA that integrates the background, and the improved DiceLoss loss function is used to enhance the distinction between the foreground and the background.

[0052] Figure 2 The details of the improvement of the ResNet network by the multi-level separation feature enhancement module are shown. Split means that after the feature map undergoes 1×1 convolution, the feature map is divided into s groups in the channel dimension, and X i Indicates that each group corresponds to an equal channel width w. The divided s groups are subjected to convolution operation Conv, batch normalization operation, and network nonlinearity is increased through Relu activation function. Represents this series of operations. i The corresponding output feature map is y i Then the feature map is divided into two subgroups, denoted as Y i,1 and Y i,2 Then Y i,2 With the next set of input X i+1 After splicing, feed to In , Concat is the connection of two feature maps in the channel dimension, using After all input feature groups are processed, the channel size will be obtained by concatenating all Y i,1 Restore, each Y i,1 All have different numbers of channels. i The calculation formula is as follows:

[0053]

[0054] The more channels included, the larger the receptive field. The output feature map in the front has fewer convolutions and a small receptive field, which is important for identifying small objects or key parts of objects; while the output feature map in the back has more convolutions connected from the front group, which can capture more large objects.

[0055] This module includes two key operations: Split and Concat. Split is used to group features, and each group has the same number of channels. The design of Split is inspired by GhostNet. The two groups of features after separation are used for identity mapping and for extracting more refined features. Concat can be divided into two types: the first is used to concatenate features through cascade y i-1,2 and x i , which fuses different features and enhances the information interaction of different groups; the second is to merge all y i,1 The Concat design is inspired by Res2Net, which uses the Summation operation, which may change or even destroy the original feature information. Therefore, using Concat instead helps to better maintain the expressiveness of the original features.

[0056] like Figure 3 As shown in Figure 1, feature maps of different scales are scaled to the same resolution after upsampling and stacking, and then input into the feature fusion module based on the attention mechanism. The input contains 4 feature maps First, the upsampled features X are concatenated, and then a 3×3 convolution is performed to obtain the intermediate features Secondly, the intermediate features are input into the spatial attention module to calculate the spatial attention weights The designed spatial attention mechanism module performs maximum pooling and average pooling operations on the processed input features, then concatenates them, and then performs a convolution on the concatenated features for fusion. The spatial attention weight is then divided into four parts along the channel dimension, and the weighted weights are multiplied by the corresponding proportional features to obtain the fused features. The calculation formula of the core stage characteristic value is as follows:

[0057] M=Concat([X0,X1,X2,X3]), (2)

[0058] SA=Sigmoid(Conv(Pool max (M)||Pool avg (M))), (3)

[0059] F=Concat([E0X0,E1x1,E2x2,E3x3]), (4)

[0060] Among them, Concat represents the cascade operator, Conv represents the convolution operator, Sigmoid is the conventional activation function, || represents splicing in the channel dimension, Pool max Represents the maximum pooling operation, Pool avg represents the average pooling operation, both of which include traditional ReLU and Sigmoid operations, E i Represents X i The corresponding proportionality factor.

[0061] like Figure 4As shown in the figure, Feature Pyramid Networks (FPN) is a module widely used for target detection and is used in many existing text detection networks based on semantic segmentation. Many past studies have shown that the feature map of the feature pyramid can capture the visual features of objects at different scales. The emergence of FPN has greatly improved the target detection performance and has become a standard component of most SOTA target detectors. PAN proposes a lightweight feature pyramid enhancement module FPEM (Feature Pyramid Enhancement Module), and uses a feature fusion module FFM (Feature Fusion Module) to fuse the entire output result. FPEM is a cascadeable module, so a considerable number of FPEMs need to be stacked to achieve the purpose of expanding the receptive field. In PAN, all FPEM results are input into the FFM module for feature fusion. The strategy used in the present invention, FPEM, consists of two stages, namely upscaling enhancement and downscaling enhancement. In the upscaling enhancement stage, the input feature maps with strides of 32, 16, 8 and 4 pixels are iteratively enhanced. In the downscaling enhancement stage, the input feature maps with strides of 4, 8, 16 and 32 pixels are reversely iterated and enhanced. The present invention uses the CARAFE upsampling operator to replace the original double interpolation upsampling algorithm, which can significantly improve the performance while only bringing negligible additional parameters and calculations. At the output stage of each module, the input of the current module is added element by element to the features generated by downscaling enhancement to obtain the result. This method is equivalent to combining FPEM and FFM, which improves efficiency.

[0062] In the text detection stage, the present invention adopts an improved pixel aggregation post-processing algorithm to predict the text box, and takes background pixels into consideration in the pixel aggregation stage, which is more adaptable to the complex environment of the factory. Figure 5 As shown in the figure, after simple concatenation and convolution operations, the output feature map generates a lightweight detection head, which is used to predict instance vectors, text regions, and text kernels. (1) The text region represents the complete shape of the text content, but the text volume in industrial labels is large and dense, resulting in close stacking of text lines and prone to overlap. (2) The text kernel represents the central area of ​​the text. Compared with the rough text region, the text kernel can better distinguish adjacent text lines. (3) The instance vector is a high-dimensional representation that contains the instance information of each pixel. It is a tool used to recover the complete text line from the text kernel. Pixels belonging to the same text line often have similar instance vectors. Therefore, the instance vector can point the pixels in the text region to the corresponding text kernel, thereby successfully combining the advantages of the text kernel of the text region.

[0063] This module combines the outputs of the above detection heads using a pixel aggregation method inspired by clustering algorithms. Figure 6 As shown in the figure, different text lines are regarded as different clusters, the text core is the cluster center, and the pixels in the text area are the samples to be clustered. In the same text line, in order to guide the pixels to the corresponding text core, the distance between the two should be minimized. In the training stage, the formula of the clustering loss function is:

[0064]

[0065]

[0066]

[0067] Where N is the number of text lines, R i represents the i-th text region, K i Represents a text line R i The corresponding kernel, Represents text pixel r and text kernel K i The distance between them, the Relu function is used to ensure that the calculation is non-negative, represents the instance vector corresponding to pixel r, Represents the text kernel K i The corresponding instance vector, λ clu is a constant with a value of 0.5.

[0068] The cluster center needs to ensure sufficient discrimination, so the instance vector of the text core needs to maintain a certain distance from the adjacent text core. On this basis, the present invention also introduces the distinction of background pixels, and the recognition loss function is as follows:

[0069]

[0070]

[0071]

[0072] Among them, P bg represents the instance vector of the background pixel, represents the distance between the text kernel and the background vector, Represents the text kernel K i With K j The distance between dis is a constant with a value of 3.

[0073] The post-processing process of pixel aggregation is as follows: (1) find the connected components in the segmentation results of the text kernel and regard each connected component as a text kernel; (2) for each text kernel K i, when the Euclidean distance of its instance vector is less than the threshold d, its neighboring pixel p is included in the corresponding text region; (3) Repeat the second step until there are no available neighboring pixels in the text region.

[0074] like Figure 5 As shown in Figure 1, the post-processing of the pixel aggregation module requires masks of the text region and text kernel, so corresponding mask labels are required during the training process. The mask label of the text region is generated by filling the ground-truth bounding box, where text pixels are 1 and non-text pixels are 0, represented by y text The text kernel generates its mask label by shrinking the original bounding boxes to a certain distance and filling them. The original bounding boxes are first shrunk using the Vatti cropping algorithm, and then the shrunk bounding boxes are converted into a binary mask, i.e., the label of the text kernel, denoted as y ker .

[0075] The total loss function can be expressed as the weighted sum of the individual part loss functions:

[0076]

[0077] in, is the loss value of the text area, is the loss value of the text kernel, See formula 5, As shown in Equation 8, α and β are used to control the balance between loss values ​​and are set to 0.5 and 0.25 respectively in the laboratory. and Considering the extreme imbalance between text pixels and non-text pixels, the present invention adopts an improved Dice Loss function for training, which is expressed as follows:

[0078]

[0079]

[0080] Among them, y text (i) is the ground truth value of the i-th pixel in the text area, p text (i) is the predicted value of the corresponding segmentation result. ker (i) is the ground truth value of the i-th pixel in the text kernel, p ker(i) is the corresponding predicted value. Dice Loss is a region-based loss function that has good performance for scenes with severe imbalance of positive and negative samples. It focuses more on mining the foreground area during training and is widely used in medical image segmentation. However, it is prone to instability during the training phase, especially in the case of small targets, which may lead to gradient saturation. Therefore, the present invention uses the improved Dice Loss based on the design of the reference focal loss function, by adding a scaling factor (1-p) θ To deal with the foreground and background balance problem during training, θ is usually set to 1.

Claims

1. A label text detection method based on multi-level separation feature enhancement and spatial attention, characterized by The method comprises the following steps: Step (1), photographing the industrial product, obtaining the industrial product label text image, and preprocessing the industrial product label text image; Step (2), put the preprocessed text image into ResNet50HS-Block to obtain a sufficient receptive field and obtain a multi-scale feature map; The ResNet50HS-Block uses the ResNet50 network as the skeleton and replaces the original 3×3 convolution module with the multi-level separation module Hierarchical-SplitBlock; Step (3), using the dual-branch design idea, the multi-scale feature map output by ResNet50HS-Block is sent to the spatial attention-based feature fusion module SAFFMG and the progressive fusion feature pyramid module CFPEM to extract multi-scale feature information; The spatial attention-based feature fusion module SAFFMG is used to extract multi-scale feature information, specifically: The feature maps of different scales output by the ResNet50HS-Block are scaled to the same resolution after upsampling and stacking operations to obtain feature maps Then the above feature map X 1 Concatenate and then perform a 3×3 convolution to obtain the intermediate features Input the intermediate feature M into the spatial attention module and calculate the spatial attention weight According to the spatial attention weight For N feature maps Weighted, get fusion features The spatial attention module performs maximum pooling and average pooling operations on the input features, then performs concat splicing, and then fuses the spliced ​​features with the original input features again, and then performs a convolution to obtain the spatial attention weights. The progressively fused feature pyramid module CFPEM includes a feature pyramid enhancement module FPEM and a feature fusion module FFM; the feature pyramid enhancement module FPEM consists of two stages, namely upscaling enhancement and downscaling enhancement; in the output stage of upscaling enhancement, the input of the current module and the features generated by downscaling enhancement are added element by element to obtain the result; the feature fusion module FFM performs feature fusion on all the results of the feature pyramid enhancement module FPEM; Step (4), after simple splicing and convolution operations, the feature maps output by the spatial attention-based feature fusion module SAFFMG and the progressive fusion feature pyramid module CFPEM are predicted by the lightweight detection head Head to obtain the corresponding instance vector, text area, and text kernel; the instance vector, text area, and text kernel are obtained by the pixel aggregation algorithm PA that fuses the background to obtain the final detection result; The pixel aggregation algorithm PA of the fusion background is specifically: Different instance vectors are regarded as different clusters, the text core is the cluster center, and the pixels in the text area are the samples to be clustered; in the same text line, in order to guide the pixels to the corresponding text core, the distance between the two should be minimized; The cluster center needs to ensure sufficient discrimination, so the instance vector of the text core needs to maintain a certain distance from the adjacent text core; on this basis, the distinction of background pixels is introduced.

2. The method according to claim 1, characterized in that The Hierarchical-Split Block in step (2) divides the received feature map into 4 groups in the channel dimension through the Split operation. Each group of feature maps X i The first group retains the original input feature map X1; the second group first performs convolution operation Conv on the input feature map X2, batch normalization operation, and increases network nonlinearity through Relu activation function to obtain Y2, and then performs Split operation on Y2 to divide it into two parts Y 2,1 and Y 2,2 , output Y 2,1 ; Group 3 will be Y 2,2 Concat with the input feature map X3, then perform convolution operation Conv on the concatenated feature map, batch normalization operation, and increase network nonlinearity through Relu activation function to obtain Y3, and then split Y3 into two parts, output Y 3,1 , and Y 3,2 Concat with the next set of input feature maps; the fourth group will be Y 3,2 Concat with the input feature map X4, then perform convolution operation Conv on the concatenated feature map, perform batch normalization operation, and increase network nonlinearity through Relu activation function; finally, concat the outputs of the four groups to obtain a multi-scale feature map; Y i The calculation formula is as follows: Since the first group does not perform the Split operation, Y 1,2 =0.

3. A label text detection system implementing the method described in any one of claims 1 to 2, characterized in that include: An image acquisition and preprocessing module, used to acquire industrial product label text images and preprocess the industrial product label text images; ResNet50HS-Block module, used to extract multi-scale feature maps from preprocessed text images; The spatial attention-based feature fusion module SAFFMG is used to extract multi-scale feature information F; The progressive fusion feature pyramid module CFPEM is used to extract multi-scale fusion feature information; The fusion module is used to concatenate and convolve the feature maps output by the spatial attention-based feature fusion module SAFFMG and the progressive fusion feature pyramid module CFPEM to obtain the final features; The lightweight detection head obtains the corresponding instance vector, text area, and text kernel based on the final feature measurement; The output module is used to obtain the final detection result by integrating the instance vector, text area, and text kernel through the pixel aggregation algorithm PA with the background.

4. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 2.

5. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 2 is implemented.

Images