Natural scene text detection method and device based on multi-level information fusion mechanism

Abstract

The application discloses a natural scene text detection method and device based on a multi-level information fusion mechanism and relates to the field of computers and artificial intelligence. The method comprises the following steps: a backbone network is used to extract the features of an image, and the extracted features are subjected to multi-module cascaded feature enhancement processing to obtain a fusion feature map containing image global context attention information and semantic segmentation information; a Faster RCNN network is used to obtain a candidate frame containing classification score values and regression parameters according to the fusion feature map; the candidate frame is subjected to preliminary screening according to the relationship between the center coordinates of the candidate frame and the center line region of a target text boundary marking frame to obtain a text candidate frame; the comprehensive values of the text candidate frame are calculated according to the classification score values; and the text candidate frame is screened according to the comprehensive values to obtain a detection result. The application uses the above method to detect the text of an image, and the accuracy of text detection is improved.

Description

Technical Field

[0001] This invention relates to the field of computers, and in particular to a method and apparatus for natural scene text detection based on a multi-level information fusion mechanism. Background Technology

[0002] Text is the primary carrier of information exchange and the dissemination of ideas. Text within images can concisely express image information, helping people better understand the images. Scene text detection, as a crucial preliminary step in text recognition, significantly impacts the accuracy of subsequent text recognition. In the intelligent manufacturing industry, the serial numbers and important attributes of many metal parts and electronic components are marked with letters or numbers on their surfaces. Applying text detection technology to systems that detect the textual attributes of metal and electronic components not only facilitates the inventory management and statistics of these components but also promotes the practical application of solutions for identifying the attributes of metal and electronic parts.

[0003] Current scene text detection systems are mainly divided into a front-end interactive interface and a back-end core text detection module. The front-end interface of this text detection system is developed using Vue.js, primarily used for image uploading and submission, and displaying detection results. The image text detection function is implemented by the back-end core text detection module. Among deep learning-based text detection methods, some are based on general object detection frameworks and improved with scene text features. However, these methods lack processing of global contextual attention information and text semantic information, resulting in a small detection range and limited detection capabilities, thus reducing the accuracy of scene text detection. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for natural scene text detection based on a multi-level information fusion mechanism. This invention improves the accuracy of text detection by performing multi-module cascaded feature enhancement processing on image features, which can fully express the global contextual attention information and semantic segmentation information of the image, and using comprehensive numerical values ​​to filter text candidate boxes to obtain detection results.

[0005] According to one aspect of the present invention, the present invention provides a natural scene text detection method based on a multi-level information fusion mechanism, comprising:

[0006] The backbone network is used to extract features from the image, and the extracted features are subjected to multi-module cascaded feature enhancement processing to obtain a fused feature map containing global contextual attention information and semantic segmentation information of the image.

[0007] Based on the fused feature map, candidate boxes containing classification scores and regression parameters are obtained using the Faster RCNN network;

[0008] The candidate boxes are initially screened based on the relationship between the center coordinates of the candidate boxes and the center line region of the target text boundary annotation box to obtain text candidate boxes;

[0009] Based on the classification score, the comprehensive value of the text candidate box is calculated.

[0010] Based on the comprehensive numerical values, the text candidate boxes are filtered to obtain the detection results.

[0011] Optionally, the step of extracting features from the image using a backbone network and performing multi-module cascaded feature enhancement processing on the extracted features to obtain a fused feature map containing global contextual attention information and semantic segmentation information includes:

[0012] The backbone network is used to extract, normalize, and perform multi-head self-attention processing on the features of the image to obtain a fused feature map;

[0013] Channel attention processing is applied to the features in the fused feature map to obtain attention weights;

[0014] The attention weights and the fused feature map are multiplied by a matrix to obtain the attention fusion feature map;

[0015] The attention fusion feature map is subjected to semantic segmentation to obtain a text saliency segmentation map;

[0016] The text saliency segmentation map is multiplied and fused with the attention fusion feature map to obtain the text segmentation map;

[0017] The text segmentation map is added to the attention fusion feature map to obtain the fusion feature map containing global contextual attention information and semantic segmentation information of the image.

[0018] Optionally, obtaining candidate boxes containing classification scores and regression parameters using the Faster R-CNN network based on the fused feature map includes:

[0019] Text suggestion boxes and text outline maps of different sizes are generated by performing text suggestion box generation and convolution processing on the fused feature maps respectively;

[0020] The suggestion boxes are aligned with the text outline sketch to obtain text suggestion candidate box features of uniform size;

[0021] Based on the text-suggested candidate box features, the Fast R-CNN network is used for classification and regression to obtain the candidate boxes containing classification scores and regression parameters.

[0022] Optionally, the step of calculating the comprehensive value of the text candidate box based on the classification score includes:

[0023] The instance score is calculated from the pixel values ​​of the text saliency segmentation map and the text outline map.

[0024] The comprehensive value of the text candidate box is calculated using a preset formula based on the instance score and the category score.

[0025] Optionally, the step of generating text proposal boxes and performing convolution processing on the fused feature map to obtain proposal boxes and text outline maps of different sizes includes:

[0026] Text suggestion boxes are generated from the fused feature map to obtain suggestion boxes of different sizes;

[0027] The proposal box is obtained by marking positive and negative samples based on the intersection-union ratio of the area of ​​the proposed box and the area of ​​the target bounding box, as well as the intersection-union ratio between positive and negative samples.

[0028] Optionally, the step of filtering the text candidate boxes based on the comprehensive numerical value to obtain the detection result includes:

[0029] The composite values ​​are sorted, and the text candidate boxes corresponding to the largest composite values ​​are selected.

[0030] Optionally, the step of performing preliminary screening of the candidate boxes based on the relationship between the center coordinates of the candidate boxes and the center line region of the target text boundary annotation box to obtain text candidate boxes includes:

[0031] Based on the shrinkage ratio, the center line region of the target text boundary annotation box is obtained using a formula. The candidate boxes are initially screened based on whether the center coordinates of the candidate boxes are located in the center line region of the target text boundary annotation box, thus obtaining text candidate boxes.

[0032] This invention provides a natural scene text detection device based on a multi-level information fusion mechanism, comprising:

[0033] The multi-module cascaded feature enhancement module is used to extract features from the image using the backbone network, and to perform multi-module cascaded feature enhancement processing on the extracted features to obtain a fused feature map containing global contextual attention information and semantic segmentation information of the image.

[0034] The classification and regression module is used to obtain candidate boxes containing classification scores and regression parameters using the Faster R-CNN network based on the fused feature map.

[0035] The preliminary screening module is used to perform preliminary screening of the candidate boxes based on the relationship between the center coordinates of the candidate boxes and the center line area of ​​the target text boundary annotation box, so as to obtain text candidate boxes;

[0036] The calculation module is used to calculate the comprehensive value of the text candidate box based on the classification score value;

[0037] The filtering module is used to filter the text candidate boxes based on the comprehensive value to obtain the detection results.

[0038] This invention provides a natural scene text detection device based on a multi-level information fusion mechanism, comprising:

[0039] Memory, used to store computer programs;

[0040] A processor is used to implement the scene text detection method as described above when executing the computer program.

[0041] The present invention provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the scene text detection method described above.

[0042] As can be seen, this invention, through multi-module cascaded feature enhancement processing of image features, enables the final fused feature map to fully represent the global contextual attention information and semantic segmentation information of the image. Furthermore, it performs preliminary screening of candidate boxes based on the relationship between the center coordinates of the candidate boxes and the centerline region of the target text boundary annotation box, reducing interference with subsequent screening of text candidate boxes. Simultaneously, it utilizes comprehensive numerical values ​​to further screen the generated text candidate boxes, improving the accuracy of the text detection results. This application also provides a natural scene text detection device based on a multi-level information fusion mechanism, possessing the aforementioned beneficial effects. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 A flowchart illustrating a natural scene text detection method based on a multi-level information fusion mechanism provided in an embodiment of the present invention;

[0045] Figure 2 This is an overall network structure diagram of a natural scene text detection module based on a multi-level information fusion mechanism provided in an embodiment of the present invention.

[0046] Figure 3 This is a structural block diagram of a natural scene text detection device based on a multi-level information fusion mechanism provided in an embodiment of the present invention.

[0047] Figure 4 This is a structural block diagram of a natural scene text detection device based on a multi-level information fusion mechanism, provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Text is the primary carrier of information exchange and the dissemination of ideas. Text within images can concisely express image information, helping people better understand the images. Scene text detection, as a crucial preliminary step in text recognition, significantly impacts the accuracy of subsequent text recognition. In the intelligent manufacturing industry, the serial numbers and important attributes of many metal parts and electronic components are marked with letters or numbers on their surfaces. Applying text detection technology to systems that detect the textual attributes of metal and electronic components not only facilitates the inventory management and statistics of these components but also promotes the practical application of solutions for identifying the attributes of metal and electronic parts.

[0050] Current scene text detection systems are mainly divided into a front-end interactive interface and a back-end core text detection module. The front-end interface of this text detection system is developed using Vue.js, primarily used for image uploading and submission, and displaying detection results. The image text detection function is implemented by the back-end core text detection module. Among deep learning-based text detection methods, some are based on general object detection frameworks and improved with scene text features. However, these methods lack processing of global contextual attention information and text semantic information, resulting in a small detection range and limited detection capabilities, thus reducing the accuracy of scene text detection.

[0051] In view of this, the present invention provides a natural scene text detection method based on a multi-level information fusion mechanism. By performing multi-module cascaded feature enhancement processing on image features, the final fused feature map can fully represent the global contextual attention information and semantic segmentation information of the image. Furthermore, the method performs preliminary screening of candidate boxes based on the relationship between the center coordinates of the candidate boxes and the center line region of the target text boundary annotation box, reducing interference to the subsequent screening of text candidate boxes. At the same time, the method uses comprehensive numerical values ​​to screen the generated text candidate boxes, thereby improving the accuracy of the text detection results.

[0052] The following is a detailed introduction; please refer to it. Figure 1 , Figure 1 A flowchart illustrating a natural scene text detection method based on a multi-level information fusion mechanism provided in this embodiment of the invention. The method may include:

[0053] Step S101: Use the backbone network to extract features from the image, and perform multi-module cascaded feature enhancement processing on the extracted features to obtain a fused feature map containing global contextual attention information and semantic segmentation information.

[0054] In this embodiment of the invention, the content of the image is not limited; it can be an image of a metal surface or an image of an electronic component surface. It should be noted that in this embodiment of the invention, a scene image can be acquired first, and then the scene image can be preprocessed, i.e., adjusted according to a preset size. The preset size can be set by the designer according to requirements or according to the specific usage scenario; this embodiment of the invention does not impose any restrictions on this.

[0055] In this embodiment of the invention, the backbone network can be a pre-trained model consisting of a network structure (ResNet50) and a Feature Pyramid Network (FPN), where ResNet50 is a feature extraction network. It should be noted that the pre-trained model can be a model trained using an image-level text recognition task, specifically a pre-trained model trained on a synthetic dataset using an image-level text recognition task. The synthetic dataset can be SynthText. The pre-trained model can include a CNN encoder and a Self-Attention decoder. The CNN encoder encodes the features of the input image, and the self-attention module in the decoder guides the backbone network to learn useful positional information and discriminative semantic abstractions for the text detection task. Furthermore, the text recognition task can provide more text stroke feature information for text detection, enhancing the backbone network's perception of text region information in the image.

[0056] In this embodiment of the invention, the fused feature map is a fused feature map containing global contextual attention information and semantic segmentation information. Specifically, the backbone network is first used to extract, normalize, and perform multi-head self-attention processing on the image features to obtain the fused feature map. Channel attention processing is then performed on the features in the fused feature map to obtain attention weights. The attention weights and the fused feature map are then matrix-multiplied to obtain an attention fused feature map. Semantic segmentation processing is then performed on the attention fused feature map to obtain a text saliency segmentation map. Simultaneously, the text saliency segmentation map is multiplied and fused with the attention fused feature map to obtain a text segmentation map. Finally, the text segmentation map and the attention fused feature map are added together to obtain the fused feature map containing global contextual attention information and semantic segmentation information. The attention fused feature map is a feature map containing attention weights, where the attention weights represent the importance of text information numerically, and the text saliency segmentation map represents the saliency of text region features in the image.

[0057] It should be noted that in the embodiments of the present invention, feature maps of different sizes extracted by the backbone network can be normalized and multi-head self-attention processing can be performed to obtain fused feature maps containing global image context information. For example, a ResNet50 network can be used to extract text image features, and the feature maps output from the second to fifth stages of the feature extraction process can be processed by FPN to obtain four feature maps of different sizes, denoted as P. i (i = 2, 3, 4, 5), where the width and height dimensions are respectively the size of the input image. The number of channels is 256, P i Specifically, it is expressed as follows:

[0058]

[0059] Where i can take values ​​of i = 2, 3, 4, 5, C i For P i The number of channels in the feature map, H i For P i The height of the feature map, W i For P i The width of the feature map. The feature map P is modified by compressing (or expanding) the number of feature map channels and expanding (or compressing) the feature map width and height. i(i = 2, 3, 4, 5) are uniformly normalized to the width and height dimensions of the P3 feature map. Specifically, according to the rules, the sizes of the P2, P4, and P5 feature maps are normalized to the size of the P3 feature map. Convolution and upsampling operations can be used on P2, P4, and P5. Specifically, for P2, 1024 convolutions with a stride of 2 and padding of 1 are used to halve its size but expand the number of channels to 4 times the original size; for P4, convolutions with a stride of 1 and padding of 0 are used to perform a 2x bilinear interpolation upsampling, and the number of convolutions is specified as 64, compressing the number of channels to 1 / 4 of the original size; for P5, convolutions with a stride of 1 and padding of 0 are used to perform a 4x bilinear interpolation upsampling, and the number of convolutions is specified as 16, compressing the number of channels to 1 / 16 of the original size. After this processing, P... i The width and height dimensions of the feature maps (i = 2, 3, 4, 5) are the same as those of the P3 feature map, with the number of channels being 1024, 256, 64, and 16 respectively, denoted as F. i (i = 2, 3, 4, 5), F i Specifically, it is expressed as follows:

[0060]

[0061]

[0062] Where i can take values ​​of i = 2, 3, 4, 5, and n can take values ​​of n. For F i (i = 2, 3, 4, 5) The feature map dimension is represented as [c i ,h i ,w i Transform its dimensions to [h] (i = 2, 3, 4, 5). i ×w i ,c i (i = 2, 3, 4, 5), thus obtaining The feature map, represented by a two-dimensional matrix, has its row dimensions representing different positional units and its column dimensions representing the features contained within those units. The feature map is obtained by transforming the dimensions according to the number of attention heads (num_head) set in the multi-head attention layer. The feature matrix unit x is obtained by partitioning. index Feature map obtained after dimensional transformation and characteristic matrix unit x index Specifically, it is expressed as follows:

[0063]

[0064]

[0065] Where i can take the value i = 2, 3, 4, 5, and index can take the value index = 1, 2, ..., num_head. In this embodiment, num_head is num_head = 8, and then for all matrix elements x index Self-attention processing is performed to obtain feature matrix units y rich in global context information. index The feature map F is obtained by splicing and linearly transforming it. i The corresponding final output F' i Finally, for all feature maps F i The corresponding output F' i The images are then stitched together to obtain a fused feature map containing global contextual information of the image.

[0066] It should be noted that, in this embodiment of the invention, after obtaining the fused feature map containing global image context information, channel attention processing is performed on the fused feature map containing global context information to obtain attention weights. The attention weights and the fused feature map are then matrix-multiplied to obtain an attention fused feature map. Semantic segmentation processing is then performed to obtain a text saliency segmentation map. This text saliency segmentation map is then multiplied and fused with the attention fused feature map to obtain a text segmentation map. Finally, the text segmentation map and the attention fused feature map are added together to obtain a fused feature map containing both global image context attention information and semantic segmentation information. For example, 3×3 and 1×1 convolutions are used to perform global semantic segmentation on the fused feature map containing attention properties. The semantic segmentation result is then processed using a softmax function and a normalized exponential function in the channel dimension. The result of the softmax function is then expressed in exponential form to obtain a text saliency segmentation map. The fused feature map containing attention properties is then matrix-multiplied with the text saliency segmentation map to obtain a feature map containing text segmentation information. Finally, the feature map containing attention properties and the feature map containing text segmentation information are added together to obtain the final fused feature map.

[0067] In this embodiment of the invention, by using a pre-trained model generated based on a text recognition training task in the backbone network, more text stroke information can be learned, improving the backbone network's ability to extract text features from the input image and performing multi-module cascaded feature enhancement processing on the feature map to obtain a fused feature map containing global contextual attention information and semantic segmentation information. Semantic segmentation processing can increase the feedback difference between foreground text features and background features, facilitating the subsequent significant extraction of foreground text features.

[0068] Step S102: Based on the fused feature map, use the Faster RCNN network to obtain candidate boxes containing classification scores and regression parameters.

[0069] In this embodiment of the invention, the target extraction network can be a Faster R-CNN network (target detection network).

[0070] Other algorithms can also be used for target extraction, and this embodiment of the invention does not impose any restrictions on this. It should be noted that the Faster R-CNN network is designed with a Region Proposal Network (RPN) to generate candidate regions. This network provides high-quality target candidate boxes. It combines anchors with different areas and aspect ratios of each pixel on the fused feature map, foreground and background scores, and regression parameters to form proposals. In the Fast R-CNN network, the proposals are further refined for classification and regression to obtain the classification score and regression parameters of the candidate boxes. The regression parameters are the difference between the candidate box and the target region. Finally, the proposals and regression parameters are combined to obtain the final text candidate boxes.

[0071] In this embodiment of the invention, text proposal boxes and text outline maps of different sizes can be generated and convolutionally processed on the fused feature map. Then, the text proposal boxes are combined with the text outline maps to perform feature alignment processing, resulting in text proposal candidate box features of uniform size. Finally, based on the text proposal candidate box features, a Fast R-CNN network is used to obtain text candidate boxes containing classification scores and regression parameters. It should be noted that the feature alignment method used in this embodiment of the invention is a Space Transform Network (SPTNet). For example, in the feature alignment stage of the RoIAlign region, according to the uniform output feature map size a×b, the feature block corresponding to the original image is divided into a×b blocks, where a and b are the height and width, respectively. Within each block, k×k sampling points are sampled according to the sampling rate (k is the division interval). For sampling points within each block that might contribute background noise, SPTNet is used to perform spatial transformations to ensure that the transformed sampling point positions are located as close as possible to the text region. Then, bilinear interpolation is used to obtain the positional value of each sampling point. Finally, the maximum value among k×k sampling points is selected as the feature value of the block. This process yields text proposal candidate box features of uniform size. Specifically, the spatial transformation using SPTNet involves defining pixel values ​​of the text region in the outline of the text abstract contour as 1, and pixel values ​​of non-text regions as 0. For sampling points located in non-text regions, SPTNet learns the corresponding affine transformation matrix M using the 6-dimensional parametric features contained in the candidate box containing the sampling point. AT Then, based on the obtained affine transformation matrix M AT The sampling points are standardized point by point using the following formula:

[0072]

[0073] Among them, (x t ,y t (x0, y0) represents the coordinates of the normalized sampling point. Let be the affine transformation matrix.

[0074] It should be noted that in this embodiment of the invention, text proposal boxes of different sizes can be generated from the fused feature map. This can be achieved by first generating text proposal boxes from the fused feature map, and then allocating positive and negative samples according to the intersection-union ratio (IU) of the region area of ​​the proposal box and the area of ​​the target bounding box, as well as the IU between positive and negative samples, to obtain proposal boxes of different sizes. Specifically, for example, a large number of text proposal boxes generated in the RPN stage can be divided into positive and negative samples based on the IU of the text proposal boxes and the target region, and the IU between positive and negative samples. It should be noted that for negative sample sampling, the interval can be divided according to the IU value between positive and negative samples, and the interval can be divided into v equal grids. Then, samples are uniformly sampled from the corresponding intervals. The sampling interval is divided into v equal intervals, and the corresponding number of samples N is selected in each interval. v The corresponding sampling probability is obtained. Where N represents the IOU (Intersection over Union) value between positive and negative samples. Similarly, the same sampling method is used for positive samples as for negative samples. For each target region, the best matching positive sample is found, and its weight is multiplied by a coefficient during the calculation of sampling loss. Positive samples with different IOUs with the target region are reweighted and participate in training in an unequal manner, so that the obtained positive samples have a high degree of overlap with negative samples, thereby improving the accuracy of text detection.

[0075] It should be noted that, in the embodiments of the present invention, the fused feature map can be processed by convolution to obtain a text outline sketch map. Specifically, for example, the fused feature map can be processed by 1×K and K×1 convolution (K is the convolution kernel size) to obtain the features of the fused feature map in two orthogonal directions, horizontal and vertical. The feature points with strong unidirectional characteristics in two of these directions are selected as the outline map of the text abstract contour.

[0076] In this embodiment of the invention, text suggestion boxes and text contour maps of different sizes are obtained by performing text suggestion box generation and convolution processing on the fused feature map respectively. Then, the suggestion boxes are combined with the text contour maps to perform feature alignment processing to obtain text suggestion candidate box features of uniform size. Finally, based on the text suggestion candidate box features, a Fast R-CNN network is used to obtain candidate boxes containing classification score values ​​and coordinate regression parameters. This method can avoid introducing background noise and improve the accuracy of text detection.

[0077] Step S103: Perform preliminary screening of candidate boxes based on the relationship between the center coordinates of the candidate boxes and the center line area of ​​the target text boundary annotation box to obtain text candidate boxes.

[0078] In this embodiment of the invention, since the candidate bounding box of a specific text target has a large Intersection over Union (IOU) with the target text boundary annotation box, and the center coordinates of the candidate bounding box are located in the centerline region of the target text boundary annotation box, the centerline region of the target text boundary annotation box can be obtained using a formula based on the shrinkage ratio. The candidate bounding boxes are then preliminarily screened based on whether their center coordinates are located in the centerline region of the target text boundary annotation box to obtain text candidate bounding boxes. Specifically, the centerline region box' of the target text boundary annotation box is obtained by shrinking the annotation box box by q pixels. The distance q between the annotation box box and the shrunk centerline region box' is calculated using the following formula:

[0079] q = Area(box) × (1-r) 2 Perimeter (box)

[0080] Where Area(box) is the function for calculating the area of ​​the bounding box, Perimeter(box) is the function for calculating the perimeter of the bounding box, and r is the shrinkage ratio, which is r = 0.5 in this embodiment. It should be noted that if the center coordinates (x, y) of a candidate box are located within the center line area of ​​the target text bounding box, it is retained; otherwise, the candidate box's classification score is set to zero and it is discarded without further calculation of the instance score.

[0081] In this invention, candidate boxes are initially screened by checking whether their center coordinates are located within the center line region of the target text boundary annotation box. This process reduces interference with subsequent screening of candidate boxes and improves the efficiency of text detection.

[0082] Step S104: Calculate the comprehensive value of the text candidate boxes based on the classification score. After obtaining the comprehensive value, proceed to step S105: Filter the text candidate boxes based on the comprehensive value to obtain the detection results.

[0083] In this embodiment of the invention, the comprehensive score can be calculated using a preset formula based on the instance score and the category score, and is used to determine the accuracy of the text candidate box. The specific preset formula is as follows:

[0084]

[0085] Among them, S C S represents the classification score. IS' is the instance score value, and S' is the composite value. It should be noted that in this embodiment of the invention, the instance score value can be calculated from the pixel values ​​in the text saliency segmentation map and the text outline sketch map. Specifically, the instance score value S... I The pixel values ​​in the text saliency segmentation map and the text outline map can be calculated using the following formula:

[0086]

[0087] Where, ρ j δ represents the pixel value corresponding to this point in the text saliency segmentation map. j Y represents the pixel value corresponding to that point in the text outline sketch, and Y represents the number of pixels corresponding to the text suggestion box.

[0088] In this embodiment of the invention, after obtaining the comprehensive value, the text candidate boxes can be filtered according to the comprehensive value to obtain the detection result. Specifically, the comprehensive values ​​can be sorted, and then the text candidate box corresponding to the largest comprehensive value can be filtered as the detection result.

[0089] In this embodiment of the invention, a more accurate text detection result can be generated by combining the instance score value with the classification score value to form a comprehensive score, and then filtering the generated text candidate boxes based on the comprehensive score.

[0090] Based on the above embodiments, in this embodiment of the invention, multi-module cascaded feature enhancement processing is performed on image features, so that the final fused feature map can fully represent the global contextual attention information and semantic segmentation information of the image. Furthermore, the candidate boxes are initially screened based on the relationship between the center coordinates of the candidate boxes and the center line region of the target text boundary annotation box, reducing interference to the subsequent screening of text candidate boxes. At the same time, comprehensive numerical values ​​are used to screen the generated text candidate boxes, thereby improving the accuracy of the text detection results.

[0091] The above process is illustrated with specific examples below. Please refer to them. Figure 2 , Figure 2 The overall network structure diagram of a natural scene text detection module based on a multi-level information fusion mechanism provided in this embodiment of the invention is as follows:

[0092] 1. Acquire the image and extract features using a backbone network (RseNet50+FPN). Then, perform feature enhancement processing on the extracted features using a multi-module cascaded module to obtain a fused feature map containing global contextual attention information and semantic segmentation information. ResNet50 is the feature extraction network, and FPN (Feature Pyramid Network) is a feature map pyramid network to fuse features of different levels and sizes. It should be noted that the feature enhancement processing performed by the multi-module cascaded module includes:

[0093] (1) Normalize (Scale transfer) and multi-head self-attention (MHA) on the image features extracted by the backbone network, obtain the fused feature map through channel connection operation (C), and obtain the attention weights by channel attention.

[0094] (2) Multiply the attention weights and the fused feature map by matrix to obtain the attention fused feature map;

[0095] (3) Perform semantic segmentation on the attention fusion feature map to obtain the text saliency segmentation map;

[0096] (4) Multiply and fuse the text saliency segmentation map with the attention fusion feature map to obtain the text segmentation map;

[0097] (5) The text segmentation map and the attention fusion feature map are added and fused to obtain the fusion feature map containing global contextual attention information and semantic segmentation information of the image.

[0098] 2. Based on the fused feature map, the Faster R-CNN network is used to obtain candidate boxes containing classification scores and regression parameters. The specific steps for obtaining candidate boxes include:

[0099] (1) Perform 1×K and K×1 convolution processing on the fused feature map (K is the convolution kernel size) to obtain a text outline map;

[0100] (2) Text suggestion boxes are generated from the fused feature map during the RPN stage to obtain suggestion boxes of different sizes;

[0101] (3) Use the Spatial Transformation Network (SPTNet) in conjunction with the text outline map to perform feature alignment processing on the proposal box in the RoIAlign region to obtain text proposal candidate box features of uniform size;

[0102] (4) Based on the text-suggested candidate box features, the FastRCNN network is used to perform classification and regression to obtain candidate boxes containing classification scores and regression parameters.

[0103] 3. Based on the relationship between the center coordinates of the candidate boxes and the center line area of ​​the target text boundary annotation box, the candidate boxes are initially screened to obtain text candidate boxes.

[0104] 4. Based on the classification score, calculate the comprehensive value of the text candidate boxes.

[0105] 5. Based on the comprehensive numerical values, the text candidate boxes are filtered to obtain the detection results.

[0106] In this embodiment of the invention, a newly designed multi-module cascaded feature enhancement module is used in the image feature extraction and encoding stage to perform multi-angle information enhancement processing on image features, resulting in a fused feature map containing global contextual attention information and semantic segmentation information. Furthermore, the SPTNet RoIAlign network is used to learn an affine transformation matrix by combining the sampling point position information from the feature alignment stage with the text contour map generated by horizontal and vertical convolutions. This constrains the sampling point positions to ensure they are located as close as possible to the text region, avoiding the introduction of background noise into the candidate box suggestion features. Additionally, the candidate boxes are initially screened based on the relationship between the center coordinates of the candidate boxes and the centerline region of the target text boundary annotation box, reducing interference with subsequent text candidate box screening. Finally, the text saliency segmentation map generated in the feature enhancement stage and the corresponding pixel values ​​in the text contour map generated by horizontal and vertical convolutions are combined to form an instance score. This instance score is then combined with the classification score obtained from the final candidate boxes to form a new comprehensive score for screening the generated text candidate boxes, resulting in more accurate text detection results.

[0107] The following describes a natural scene text detection device and apparatus based on a multi-level information fusion mechanism provided by an embodiment of the present invention. The natural scene text detection device and apparatus based on a multi-level information fusion mechanism described below can be referred to in correspondence with the natural scene text detection method based on a multi-level information fusion mechanism described above.

[0108] Please refer to Figure 3 , Figure 3 This is a structural block diagram of a natural scene text detection device based on a multi-level information fusion mechanism provided in an embodiment of the present invention. The device may include:

[0109] The multi-module cascaded feature enhancement module 10 is used to extract features from the image using the backbone network and perform multi-module cascaded feature enhancement processing on the extracted features to obtain a fused feature map containing global contextual attention information and semantic segmentation information of the image.

[0110] The classification and regression module 20 is used to obtain candidate boxes containing classification scores and regression parameters using the Faster R-CNN network based on the fused feature map.

[0111] The preliminary screening module 30 is used to perform preliminary screening of the candidate boxes based on the relationship between the center coordinates of the candidate boxes and the center line area of ​​the target text boundary annotation box, so as to obtain text candidate boxes;

[0112] The calculation module 40 is used to calculate the comprehensive value of the text candidate box based on the classification score value;

[0113] The filtering module 50 is used to filter the text candidate boxes according to the comprehensive value to obtain the detection result.

[0114] Based on the above embodiments, the multi-module cascaded feature enhancement module 10 may include:

[0115] The normalization and multi-head self-attention processing unit is used to extract, normalize, and perform multi-head self-attention processing on the features of the image using the backbone network to obtain a fused feature map.

[0116] In this embodiment of the invention, multi-head self-attention processing can be performed on each feature map of the same size but different levels, and the processing results of different feature layers can be stitched together to obtain a fused feature map containing global image context information.

[0117] The channel attention processing unit is used to perform channel attention processing on the features in the fused feature map to obtain attention weights, and to perform matrix multiplication of the attention weights and the fused feature map to obtain the attention fused feature map;

[0118] The text semantic segmentation processing unit is used to perform semantic segmentation processing on the attention fusion feature map to obtain a text saliency segmentation map, and then multiply and fuse the text saliency segmentation map with the attention fusion feature map to obtain a feature map containing text segmentation information.

[0119] The fusion unit is used to add the attention fusion feature map and the feature map containing text segmentation information to obtain a fusion feature map containing global contextual attention information and semantic segmentation information.

[0120] Based on any of the above embodiments, the classification and regression module 20 may include:

[0121] The processing unit is used to perform text suggestion box generation and convolution processing on the fused feature map to obtain suggestion boxes and text outline maps of different sizes;

[0122] In this embodiment of the invention, the fused feature map can be processed by 1×K and K×1 convolution (K is the size of the convolution kernel) to obtain the features of the fused feature map in two orthogonal directions, horizontal and vertical. The feature points with strong unidirectional characteristics in two of these directions are selected as the approximate outline of the text abstract contour. SPTNet can be used to transform the spatial position of sampling points in sub-region units with uniform output feature map size that may bring background noise, so that the transformed sampling point positions are located in the text region as much as possible. Then, bilinear interpolation is used to obtain the position value of each sampling point, and then the maximum value of the sampling points in the sub-region unit is selected as the feature value of the sub-region.

[0123] The feature alignment unit is used to perform feature alignment processing on the suggestion box in conjunction with the text outline sketch to obtain text suggestion candidate box features of uniform size;

[0124] The classification and regression processing unit is used to classify and regress the candidate boxes using the FastRCNN network based on the text suggestion candidate box features to obtain the candidate boxes containing classification scores and regression parameters.

[0125] Based on any of the above embodiments, the processing unit may include:

[0126] The text suggestion box generation subunit is used to generate text suggestion boxes from the fused feature map to obtain suggestion boxes of different sizes.

[0127] The tagging subunit is used to tag positive and negative samples to obtain the proposal box based on the intersection-union ratio of the area of ​​the proposal box and the area of ​​the target bounding box, as well as the intersection-union ratio between positive and negative samples.

[0128] Based on any of the above embodiments, the preliminary screening module 40 may include:

[0129] The judgment unit is used to obtain the center line region of the target text boundary annotation box according to the shrinkage ratio and the formula, and to perform preliminary screening of the candidate boxes based on whether the center coordinates of the candidate boxes are located in the center line region of the target text boundary annotation box, so as to obtain the text candidate boxes.

[0130] In this embodiment of the invention, the distance pixel q between the annotation box and the center line area of ​​the annotation box can be calculated according to a specified formula. It can also determine whether the center coordinates (x, y) of the candidate box are located in the center line area of ​​the target text boundary annotation box, and then remove the candidate boxes that do not meet the conditions, without calculating the subsequent instance score value, thus reducing the interference to the subsequent screening of text candidate boxes.

[0131] Based on any of the above embodiments, the computing module 50 may include:

[0132] The instance score calculation unit is used to calculate the instance score value from the pixel values ​​in the text saliency segmentation map and the text outline map.

[0133] The comprehensive numerical calculation unit is used to calculate the comprehensive value of the text candidate box using a preset formula based on the classification score value and the instance score value.

[0134] Based on any of the above embodiments, the filtering module 60 may include:

[0135] A sorting unit is used to sort the comprehensive values;

[0136] The filtering unit is used to filter out the text candidate boxes corresponding to the largest comprehensive value.

[0137] In this embodiment of the invention, multi-module cascaded feature enhancement processing is performed on image features to obtain a fused feature map containing global contextual attention information and semantic segmentation information. The candidate boxes are initially screened based on the relationship between the center coordinates of the candidate boxes and the center line region of the target text boundary annotation box, reducing interference to the subsequent screening of text candidate boxes. At the same time, a comprehensive value is calculated based on the classification score and instance score, and the comprehensive value is used to screen the generated text candidate boxes, thereby improving the accuracy of the text detection results.

[0138] Please refer to Figure 4 , Figure 4 This is a structural block diagram of a natural scene text detection device based on a multi-level information fusion mechanism provided in an embodiment of the present invention. The device includes:

[0139] Memory 10 is used to store computer programs;

[0140] When the processor 20 executes the computer program, it implements the above-described natural scene text detection method based on a multi-level information fusion mechanism.

[0141] like Figure 4 The diagram shown is a structural schematic of a computer device, which may include: a memory 10, a processor 20, a communication interface 31, an input / output interface 32, and a communication bus 33.

[0142] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 10 may store programs for implementing the following functions:

[0143] The backbone network is used to extract features from the image, and the extracted features are then subjected to multi-module cascaded feature enhancement processing to obtain a fused feature map containing global contextual attention information and semantic segmentation information.

[0144] Based on the fused feature map, candidate boxes containing classification scores and regression parameters are obtained using the Faster R-CNN network;

[0145] The candidate boxes are initially screened based on the relationship between the center coordinates of the candidate boxes and the center line area of ​​the target text boundary annotation box to obtain text candidate boxes;

[0146] Based on the classification score, the comprehensive value of the text candidate boxes is calculated.

[0147] Based on the comprehensive numerical values, the text candidate boxes are filtered to obtain the detection results.

[0148] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0149] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0150] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0151] The communication interface 31 can be an interface for connecting with other devices or systems.

[0152] The input / output interface 32 can be an interface used to acquire external input data or output data to the outside world.

[0153] Of course, it should be noted that, Figure 4 The structure shown does not constitute a limitation on the computer device in the embodiments of this application. In practical applications, the computer device may include more than Figure 4More or fewer components as shown, or combinations of certain components.

[0154] In this embodiment of the invention, multi-module cascaded feature enhancement processing is performed so that the final fused feature map can fully represent the global contextual attention information and semantic segmentation information of the image. The candidate boxes are initially screened based on the relationship between the center coordinates of the candidate boxes and the center line region of the target text boundary annotation box, which reduces the interference to the subsequent screening of text candidate boxes. At the same time, a comprehensive value is calculated based on the classification score and instance score, and the comprehensive value is used to screen the generated text candidate boxes, thereby improving the accuracy of the text detection results.

[0155] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these instructions are loaded and executed by a processor, they enable the extraction of image features using a backbone network, followed by multi-module cascaded feature enhancement processing to obtain a fused feature map containing global contextual attention information and semantic segmentation information. Based on the fused feature map, a Faster R-CNN network is used to obtain candidate boxes containing classification scores and regression parameters. The candidate boxes are initially screened based on the relationship between their center coordinates and the centerline region of the target text boundary annotation box to obtain text candidate boxes. A comprehensive value for the text candidate boxes is calculated based on the classification scores. Finally, the text candidate boxes are further screened based on the comprehensive value to obtain the detection results. In this embodiment of the invention, multi-module cascaded feature enhancement processing is performed on image features, so that the final fused feature map can fully represent the global contextual attention information and semantic segmentation information of the image. The candidate boxes are initially screened based on the relationship between the center coordinates of the candidate boxes and the center line region of the target text boundary annotation box, which reduces the interference to the subsequent screening of text candidate boxes. At the same time, a comprehensive value is calculated based on the classification score and instance score, and the comprehensive value is used to screen the generated text candidate boxes, thereby improving the accuracy of the text detection results.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0157] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0158] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0159] The foregoing has provided a detailed description of the natural scene text detection method, apparatus, computer device, and storage medium based on a multi-level information fusion mechanism provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A method for natural scene text detection based on a multi-level information fusion mechanism, characterized in that, include: The backbone network is used to extract features from the image, and the extracted features are subjected to multi-module cascaded feature enhancement processing to obtain a fused feature map containing global contextual attention information and semantic segmentation information of the image. Based on the fused feature map, candidate boxes containing classification scores and regression parameters are obtained using the Faster RCNN network; The candidate boxes are initially screened based on the relationship between the center coordinates of the candidate boxes and the center line region of the target text boundary annotation box to obtain text candidate boxes; Based on the classification score, the comprehensive value of the text candidate box is calculated. Based on the comprehensive numerical values, the text candidate boxes are filtered to obtain the detection results; The process of extracting image features using a backbone network and then performing multi-module cascaded feature enhancement on the extracted features to obtain a fused feature map containing global contextual attention information and semantic segmentation information includes: The backbone network is used to extract, normalize, and perform multi-head self-attention processing on the features of the image to obtain a fused feature map; Channel attention processing is applied to the features in the fused feature map to obtain attention weights; The attention weights and the fused feature map are multiplied by a matrix to obtain the attention fusion feature map; The attention fusion feature map is subjected to semantic segmentation to obtain a text saliency segmentation map; The text saliency segmentation map is multiplied and fused with the attention fusion feature map to obtain the text segmentation map; The text segmentation map is added to the attention fusion feature map to obtain the fusion feature map containing global image context attention information and semantic segmentation information; The step of obtaining candidate boxes containing classification scores and regression parameters using the Faster R-CNN network based on the fused feature map includes: Text suggestion boxes and text outline maps of different sizes are generated by performing text suggestion box generation and convolution processing on the fused feature maps respectively; The suggestion boxes are aligned with the text outline sketch to obtain text suggestion candidate box features of uniform size; Based on the text-suggested candidate box features, the FastRCNN network is used for classification and regression to obtain the candidate boxes containing classification scores and regression parameters; The preliminary screening of candidate boxes based on the relationship between the center coordinates of the candidate boxes and the center line region of the target text boundary annotation box, to obtain text candidate boxes, includes: Based on the shrinkage ratio, the center line region of the target text boundary annotation box is obtained using a formula. The candidate boxes are initially screened based on whether the center coordinates of the candidate boxes are located in the center line region of the target text boundary annotation box, and text candidate boxes are obtained. The formula is: ； in, A function to calculate the area of ​​the bounding box of the target text. A function to calculate the perimeter of the bounding box of the target text. This is the shrinkage ratio. , This is the distance between the target text boundary annotation box and the center line area of ​​the target text boundary annotation box. Define bounding boxes for the target text; The preliminary screening of candidate boxes based on whether their center coordinates are located within the centerline region of the target text boundary annotation box includes: If the center coordinates of the candidate box are located within the center line area of ​​the target text boundary annotation box, then the candidate box is retained; If the center coordinates of the candidate box are not located within the center line area of ​​the target text boundary annotation box, then the candidate box is removed. The step of calculating the comprehensive value of the text candidate box based on the classification score includes: The instance score is calculated from the pixel values ​​of the text saliency segmentation map and the text outline map. The comprehensive value of the text candidate box is calculated using a preset formula based on the instance score value and the classification score value. The formula for calculating the instance score is as follows: ； in, For text saliency segmentation graph The pixel value corresponding to the point. For the text outline sketch The pixel value corresponding to the point. The number of pixels corresponding to the text suggestion box; The preset formula is: ； in, This is the category score value. For example, the score value. This is a composite value.

2. The natural scene text detection method based on a multi-level information fusion mechanism as described in claim 1, characterized in that, The process of generating text proposal boxes and performing convolution processing on the fused feature map to obtain proposal boxes and text outline maps of different sizes includes: Text suggestion boxes are generated from the fused feature map to obtain suggestion boxes of different sizes; The proposal box is obtained by marking positive and negative samples based on the intersection-union ratio of the area of ​​the proposed box and the area of ​​the target bounding box, as well as the intersection-union ratio between positive and negative samples.

3. The natural scene text detection method based on a multi-level information fusion mechanism as described in claim 1, characterized in that, The step of filtering the text candidate boxes based on the comprehensive numerical value to obtain the detection results includes: The composite values ​​are sorted, and the text candidate boxes corresponding to the largest composite values ​​are selected.

4. A natural scene text detection device based on a multi-level information fusion mechanism, characterized in that, include: The multi-module cascaded feature enhancement module is used to extract features from the image using the backbone network, and to perform multi-module cascaded feature enhancement processing on the extracted features to obtain a fused feature map containing global contextual attention information and semantic segmentation information of the image. The classification and regression module is used to obtain candidate boxes containing classification scores and regression parameters using the Faster R-CNN network based on the fused feature map. The preliminary screening module is used to perform preliminary screening of the candidate boxes based on the relationship between the center coordinates of the candidate boxes and the center line area of ​​the target text boundary annotation box, so as to obtain text candidate boxes; The calculation module is used to calculate the comprehensive value of the text candidate box based on the classification score value; The filtering module is used to filter the text candidate boxes according to the comprehensive value to obtain the detection results; The multi-module cascaded feature enhancement module is specifically used to extract, normalize, and perform multi-head self-attention processing on the image features using the backbone network to obtain a fused feature map; perform channel attention processing on the features in the fused feature map to obtain attention weights; perform matrix multiplication of the attention weights and the fused feature map to obtain an attention fused feature map; perform semantic segmentation processing on the attention fused feature map to obtain a text saliency segmentation map; multiply the text saliency segmentation map and the attention fused feature map to obtain a text segmentation map; and add the text segmentation map and the attention fused feature map to obtain the fused feature map containing global contextual attention information and semantic segmentation information. The classification and regression module is specifically used to generate text suggestion boxes and perform convolution processing on the fused feature map to obtain suggestion boxes and text outline maps of different sizes; combine the text outline maps to perform feature alignment processing on the suggestion boxes to obtain text suggestion candidate box features of uniform size; and use the FastRCNN network to perform classification and regression based on the text suggestion candidate box features to obtain the candidate boxes containing classification scores and regression parameters. The preliminary screening module includes: The judgment unit is used to obtain the center line region of the target text boundary annotation box according to the shrinkage ratio and the formula, and to perform preliminary screening of the candidate boxes based on whether the center coordinates of the candidate boxes are located in the center line region of the target text boundary annotation box to obtain text candidate boxes. The formula is: ； in, A function to calculate the area of ​​the bounding box of the target text. A function to calculate the perimeter of the bounding box of the target text. This is the shrinkage ratio. , This is the distance between the target text boundary annotation box and the center line area of ​​the target text boundary annotation box. Define bounding boxes for the target text; The judgment unit is specifically used to retain the candidate box if the center coordinates of the candidate box are located within the center line area of ​​the target text boundary annotation box; and to remove the candidate box if the center coordinates of the candidate box are not located within the center line area of ​​the target text boundary annotation box. The calculation module is specifically used to calculate the instance score value by calculating the pixel value in the text saliency segmentation map and the text outline map; and to calculate the comprehensive value of the text candidate box by using a preset formula based on the instance score value and the classification score value. The formula for calculating the instance score is as follows: ； in, For text saliency segmentation graph The pixel value corresponding to the point. For the text outline sketch The pixel value corresponding to the point. The number of pixels corresponding to the text suggestion box; The preset formula is: ； in, This is the category score value. For example, the score value. This is a composite value.

5. A natural scene text detection device based on a multi-level information fusion mechanism, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the scene text detection method as described in any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the scene text detection method as described in any one of claims 1 to 3.

Images