Text color recognition method and device

Abstract

The invention relates to a text color recognition method and device. The method comprises the steps of extracting a feature map of a to-be-recognized text image; according to the feature map, text color block segmentation boundaries distributed in the text arrangement direction are determined in the to-be-recognized text image, and the text color blocks are areas with text color consistency in the to-be-recognized text image; according to the text color block segmentation boundary, the feature graph is divided into at least one feature sub-graph, and each feature sub-graph corresponds to one text color block in the to-be-recognized text image; and according to the feature sub-graph, determining the color of the text in the corresponding text color block. Therefore, the efficiency and accuracy of text color recognition can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method and device for text color recognition. Background Art

[0002] In the field of image and document processing, color restoration is a key task. Especially in applications such as text recognition and document analysis, accurately restoring text color is crucial to improving the accuracy and reliability of subsequent processing results.

[0003] Currently, there are several approaches to color restoration in image documents. The first approach involves color calculation schemes based on digital image processing. These schemes use a series of image processing steps to identify the foreground region containing text. The colors of the pixels in this foreground region are then counted, and the color with the highest frequency of occurrence is used as the predicted text color. The second approach involves color calculation schemes based on image mask prediction. These schemes use a deep learning segmentation network to predict a binary mask for the text region. This mask is then used to extract the foreground region of the text. The pixel colors within this foreground region are then counted to predict the text color.

[0004] However, the first type of method suffers from poor robustness, especially in images with complex backgrounds, resulting in poor text color detection accuracy. The second type of method overly relies on upstream foreground mask prediction results. Once the mask prediction is wrong, it will directly lead to deviations in downstream text color statistics, affecting the accuracy of overall color restoration. Summary of the Invention

[0005] The present application provides a text color recognition method and device to solve the technical problems of low accuracy and poor efficiency in the prior art when recognizing the color of text characters in text images.

[0006] In a first aspect, the present application provides a text color recognition method, the method comprising:

[0007] Extract feature maps of text images to be recognized;

[0008] Determining, based on the feature map, text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized, wherein the text color block is an area with consistent text color in the text image to be recognized;

[0009] Dividing the feature map into at least one feature sub-map according to the text color block segmentation boundary, wherein each feature sub-map corresponds to one of the text color blocks in the text image to be recognized;

[0010] The color of the text in the corresponding text color block is determined according to the characteristic sub-graph.

[0011] In a possible implementation, determining text color block segmentation boundaries distributed along a text arrangement direction in the to-be-recognized text image based on the feature map includes:

[0012] Inputting the feature map into a trained text region segmentation network to obtain prediction parameters output by the text region segmentation network, wherein the prediction parameters include confidence scores corresponding to text color block segmentation boundaries of each pixel band distributed along the text arrangement direction in the text image to be recognized, each pixel band consisting of a plurality of pixels continuously arranged in a first direction, the first direction being perpendicular to the text arrangement direction;

[0013] According to the prediction parameters, text color block segmentation boundaries distributed along the text arrangement direction are determined in the text image to be recognized.

[0014] In a possible implementation, determining text color block segmentation boundaries distributed along a text arrangement direction in the to-be-recognized text image based on the prediction parameters includes:

[0015] Determining a set of candidate pixel bands from each of the pixel bands according to the prediction parameters, wherein the confidence scores corresponding to the pixel bands in the set of candidate pixel bands are greater than or equal to a set threshold;

[0016] The text color block segmentation boundary is determined from the candidate pixel band set.

[0017] In a possible implementation, determining the text color block segmentation boundary from the candidate pixel band set includes:

[0018] The iterative process is performed according to the following steps until all pixel bands in the candidate pixel band set are marked with the first identifier:

[0019] Determining, among the pixel bands not marked with the first identifier in the candidate pixel band set, the pixel band with the highest corresponding confidence score;

[0020] Record the corresponding pixel band with the highest confidence score, and mark all pixel bands within its neighborhood window with a first identifier;

[0021] After the iterative process is completed, all recorded pixel bands are determined as text color block segmentation boundaries.

[0022] In one possible implementation, the text region segmentation network includes: a downsampling module and a classification prediction module; inputting the feature map into the trained text region segmentation network to obtain prediction parameters output by the text region segmentation network includes:

[0023] Inputting the feature map into the trained text region segmentation network, and using the downsampling module to perform layer-by-layer downsampling processing on the feature map to obtain a downsampling processing result;

[0024] The classification prediction module is used to obtain the prediction parameters based on the downsampling processing result.

[0025] In a possible implementation, the downsampling module includes N sequentially connected convolutional layers, and each of the convolutional layers performs a convolution operation in a text arrangement direction of the text image to be recognized.

[0026] In one possible implementation, extracting a feature map of the text image to be recognized includes:

[0027] generating a foreground mask image of the text image to be recognized;

[0028] The text image to be recognized and the foreground mask image are input into a trained image feature extraction network to obtain a feature map output by the image feature extraction network.

[0029] In one possible implementation, the image feature extraction network includes: a feature encoding module and a feature pyramid module; inputting the to-be-recognized text image and the foreground mask image into a trained image feature extraction network to obtain a feature map output by the image feature extraction network includes:

[0030] Inputting the to-be-recognized text image and the foreground mask image into a trained image feature extraction network, and performing multi-scale feature encoding processing on the to-be-recognized text image and the foreground mask image using the feature encoding module to obtain a plurality of feature maps of different scales;

[0031] The feature pyramid module is used to perform cross-scale feature fusion processing on the feature maps of the multiple different scales to obtain the feature map of the text image to be recognized.

[0032] In a possible implementation, the feature encoding module includes M feature extraction units connected in sequence, where M is a natural number greater than 2;

[0033] The step of performing multi-scale feature extraction processing on the to-be-recognized text image and the foreground mask image by using the feature encoding module includes:

[0034] Using the first feature extraction unit to perform feature extraction processing on the text image to be recognized and the foreground mask image to obtain a feature map at a first scale;

[0035] The i-th feature extraction unit is used to perform feature extraction processing on the feature map at the i-1-th scale to obtain a feature map at the i-th scale, where i is an integer in the range of [2, M].

[0036] In a possible implementation, the feature pyramid module includes M-1 sequentially connected up-sampling units, and the cross-scale feature fusion processing of the feature maps at different scales by using the feature pyramid module includes:

[0037] The first down-sampling unit is used to perform feature fusion processing on the feature map at the M-th scale and the feature map at the M-1-th scale to obtain a first feature fusion result;

[0038] The j-th down-sampling unit is used to perform feature fusion processing on the (j-1)-th feature fusion result and the feature map at the s-th scale to obtain a j-th feature fusion result, where j is an integer in the range of [2, M-1], and s is an integer in the range of [M-2, 1];

[0039] The (M-1)-th feature fusion result is determined as the feature map of the text image to be recognized.

[0040] In a possible implementation, the color of the text in the corresponding text color block is determined according to the feature subgraph, including:

[0041] The feature subgraph is input into a trained color regression prediction network to obtain the color of the text in the corresponding text color block of the feature subgraph.

[0042] In a possible implementation, the color regression prediction network includes a feature pooling layer, a highlight RGB regression module, and a text RGB regression module.

[0043] The feature subgraph is input into a trained color regression prediction network to obtain the color of the text in the corresponding text color block of the feature subgraph, including:

[0044] The feature subgraph is input into a trained color regression prediction network, and the feature subgraph is subjected to feature extraction processing by using the feature pooling layer to obtain a feature vector of the corresponding text color block.

[0045] The highlight RGB regression module is used to extract an RGB regression prediction result of a highlight region in the corresponding text color block from the feature vector, and the text RGB regression module is used to extract an RGB regression prediction result of a text region in the corresponding text color block from the feature vector.

[0046] The color of the text in the text color block is determined according to the RGB regression prediction result of the highlight area in the text color block and the RGB regression prediction result of the text area.

[0047] In a second aspect, the present application provides a text color recognition device, the device comprising:

[0048] A feature extraction module is used to extract feature maps of the text image to be recognized;

[0049] a segmentation boundary determination module, configured to determine, in the text image to be recognized, text color block segmentation boundaries distributed along a text arrangement direction based on the feature map, wherein the text color block is an area with consistent text color in the text image to be recognized;

[0050] a segmentation module, configured to divide the feature map into at least one feature sub-map according to the text color block segmentation boundary, wherein each feature sub-map corresponds to one of the text color blocks in the text image to be recognized;

[0051] The color recognition module is used to determine the color of the text in the corresponding text color block according to the characteristic sub-image.

[0052] In a possible implementation, the segmentation boundary determination module includes:

[0053] a prediction unit, configured to input the feature map into a trained text region segmentation network to obtain prediction parameters output by the text region segmentation network, wherein the prediction parameters include confidence scores corresponding to text color block segmentation boundaries of each pixel band distributed along the text arrangement direction in the to-be-recognized text image, each pixel band consisting of a plurality of pixels continuously arranged in a first direction, the first direction being perpendicular to the text arrangement direction;

[0054] The boundary determination unit is used to determine the text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized based on the prediction parameters.

[0055] In a possible implementation manner, the boundary determination unit is specifically configured to:

[0056] Determining a set of candidate pixel bands from each of the pixel bands according to the prediction parameters, wherein the confidence scores corresponding to the pixel bands in the set of candidate pixel bands are greater than or equal to a set threshold;

[0057] The text color block segmentation boundary is determined from the candidate pixel band set.

[0058] In a possible implementation manner, the boundary determination unit is specifically configured to:

[0059] Perform iterative processing according to the following steps until all pixel bands in the candidate pixel band set are marked with the first identifier:

[0060] Determining, among the pixel bands not marked with the first identifier in the candidate pixel band set, the pixel band with the highest corresponding confidence score;

[0061] Record the corresponding pixel band with the highest confidence score, and mark all pixel bands within its neighborhood window with a first identifier;

[0062] After the iterative process is completed, all recorded pixel bands are determined as text color block segmentation boundaries.

[0063] In one possible implementation, the text region segmentation network includes: a downsampling module and a classification prediction module; the prediction unit is specifically configured to:

[0064] Inputting the feature map into the trained text region segmentation network, and using the downsampling module to perform layer-by-layer downsampling processing on the feature map to obtain a downsampling processing result;

[0065] The classification prediction module is used to obtain the prediction parameters based on the downsampling processing result.

[0066] In a possible implementation, the downsampling module includes N sequentially connected convolutional layers, and each of the convolutional layers performs a convolution operation in a text arrangement direction of the text image to be recognized.

[0067] In a possible implementation, the feature extraction module includes:

[0068] a foreground mask generating unit, configured to generate a foreground mask image of the text image to be recognized;

[0069] The extraction unit is used to input the text image to be recognized and the foreground mask image into a trained image feature extraction network to obtain a feature map output by the image feature extraction network.

[0070] In one possible implementation, the image feature extraction network includes: a feature encoding module and a feature pyramid module; the extraction unit includes:

[0071] a multi-scale extraction subunit, configured to input the to-be-recognized text image and the foreground mask image into a trained image feature extraction network, and perform multi-scale feature encoding processing on the to-be-recognized text image and the foreground mask image using the feature encoding module to obtain feature maps of multiple scales;

[0072] a fusion subunit configured to perform cross-scale feature fusion processing on the plurality of feature maps of different scales by using the feature pyramid module to obtain a feature map of the text image to be recognized.

[0073] In a possible implementation, the feature encoding module includes M sequentially connected feature extraction units, where M is a natural number greater than 2.

[0074] The multi-scale extraction subunit is specifically configured to:

[0075] The first feature extraction unit is configured to perform feature extraction processing on the text image to be recognized and the foreground mask image to obtain a feature map at a first scale.

[0076] The i th feature extraction unit is configured to perform feature extraction processing on the feature map at the (i-1) th scale to obtain a feature map at an i th scale, where i is an integer in the range of [2, M] and is taken one by one.

[0077] In a possible implementation, the feature pyramid module includes M-1 sequentially connected up-sampling units, and the fusion subunit is specifically configured to:

[0078] The first up-sampling unit is configured to perform feature fusion processing on the feature map at the M th scale and the feature map at the (M-1) th scale to obtain a first feature fusion result.

[0079] The j th up-sampling unit is configured to perform feature fusion processing on the (j-1) th feature fusion result and the feature map at the s th scale to obtain a j th feature fusion result, where j is an integer in the range of [2, M-1] and is taken one by one, and s is an integer in the range of [M-2, 1] and is taken one by one.

[0080] The (M-1) th feature fusion result is determined as the feature map of the text image to be recognized.

[0081] In a possible implementation, the color recognition module is specifically configured to:

[0082] The feature sub-map is input into the trained color regression prediction network to obtain the color of the text block in the feature sub-map.

[0083] In a possible implementation, the color regression prediction network includes a feature pooling layer, a highlight RGB regression module, and a text RGB regression module.

[0084] The feature sub-map is input into the trained color regression prediction network to obtain the color of the text block in the feature sub-map.

[0085] Inputting the feature subgraph into the trained color regression prediction network, performing feature extraction processing on the feature subgraph using the feature pooling layer to obtain a feature vector of the corresponding text color block;

[0086] Extracting the RGB regression prediction result of the highlight area in the corresponding text color block from the feature vector using the highlight RGB regression module, and extracting the RGB regression prediction result of the text area in the corresponding text color block from the feature vector using the text RGB regression module;

[0087] The color of the text in the text color block is determined according to the RGB regression prediction result of the highlight area in the text color block and the RGB regression prediction result of the text area.

[0088] In a third aspect, the present application provides an electronic device comprising: a processor and a memory, wherein the processor is configured to execute a text color recognition program stored in the memory to implement the text color recognition method described in any one of the first aspects.

[0089] In a fourth aspect, the present application provides a storage medium storing one or more programs, which can be executed by one or more processors to implement the text color recognition method described in any one of the first aspects.

[0090] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application, by extracting the feature map of the text image to be recognized, and determining the text color block segmentation boundary distributed along the text arrangement direction in the text image to be recognized according to the feature map, realizes the local consistency feature of the text color directly parsed from the feature level, can more accurately divide the text color block boundary, and avoids the detection deviation caused by background interference or mask error in the traditional method; by dividing the feature map of the text image to be recognized into at least one feature sub-map according to the text color block segmentation boundary, and determining the color of the text in the corresponding text color block according to the feature sub-map, realizes color prediction based on the entire color block, without the need to process each text character separately, so that when facing multi-color text, it can extract color information more efficiently, avoid the inefficiency of word-by-word processing, and ensure the accuracy of color prediction. In short, the method provided by the embodiment of the present application realizes end-to-end text color recognition, avoids dependence on upstream masks, and does not require additional post-processing steps, thereby significantly simplifying the calculation process, improving processing efficiency, and significantly improving the accuracy of text color recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0092] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0093] One or more embodiments are illustrated by the pictures in the drawings corresponding thereto, and these illustrative embodiments do not constitute a limitation on the embodiments, and elements with the same reference numerals in the drawings represent similar elements, unless otherwise specified. The drawings in the drawings do not constitute a proportional limitation.

[0094] Figure 1 An embodiment flow chart of a text color recognition method provided by the embodiments of the present application;

[0095] Figure 2 An example of a text image to be recognized;

[0096] Figure 3 A structural schematic diagram of a color regression prediction network;

[0097] Figure 4 An embodiment flow chart of another text color recognition method provided by the embodiments of the present application;

[0098] Figure 5 A structural schematic diagram of a foreground segmentation network;

[0099] Figure 6 A structural schematic diagram of a feature encoding module;

[0100] Figure 7 A schematic diagram of a feature encoding module;

[0101] Figure 8 An embodiment flow chart of another text color recognition method provided by the embodiments of the present application;

[0102] Figure 9 A process schematic diagram of determining a text color block segmentation boundary from a candidate pixel band set;

[0103] Figure 10 A whole structural schematic diagram of a text color recognition model provided by the embodiments of the present application;

[0104] Figure 11 An embodiment block diagram of a text color recognition device provided by the embodiments of the present application;

[0105] Figure 12 A structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0106] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0107] The disclosure below provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, these are merely examples and are not intended to limit the present application. In addition, the present application may repeat reference numbers and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0108] In order to solve the technical problems of low accuracy and poor efficiency faced by the existing technology when recognizing the color of text characters in text images, especially when dealing with complex backgrounds and multi-color text images, the present application provides a text color recognition method and device that can significantly improve the efficiency and accuracy of text color recognition.

[0109] Figure 1 This is a flow chart of an embodiment of a text color recognition method provided in an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0110] Step 101: Extract a feature map of the text image to be recognized.

[0111] The text image to be recognized refers to an image whose color of the text characters contained therein needs to be recognized. Such an image includes text characters arranged in a single row or a single column. For example, see Figure 2 , is an example of a text image to be recognized. Figure 2 The text image to be recognized shown includes text characters arranged in a single row.

[0112] In practical applications, the text image to be recognized is usually specified by the user, and its source includes two types of document forms: one is a text document, such as a Word document, a PDF document, etc. For this type of document, the original document image can be obtained by taking a photo or scanning, and then the character detection technology (such as a character detection model based on deep learning) is used to detect the bounding box of each text character from the original document image, and the character bounding boxes of the same row or column are sequentially spliced ​​into a continuous area. Finally, based on the splicing result, the original document image is cut into image blocks of multiple rows or columns, and each image block constitutes a text image to be recognized. The second is an image document, such as an image in a format such as JPEG and PNG. For this type of document, the image segmentation technology can be used to directly capture the area containing a single row or column of text characters, and the area can be cut out from the original image as the text image to be recognized.

[0113] The feature map of the text image to be recognized is a multidimensional feature representation of the text image to be recognized. Its core value lies in the in-depth analysis and reconstruction of the original image information. In one embodiment, a deep learning model can be used to extract structured features from the text image to be recognized, thereby obtaining the feature map of the text image to be recognized. By extracting the feature map of the text image to be recognized, it provides a foundation for the subsequent text character color recognition task.

[0114] As for how to extract the feature map of the text image to be recognized, we will use Figure 4 The embodiments shown are only for explanation and will not be described in detail here.

[0115] Step 102: Determine text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized based on the above-mentioned feature map, wherein the text color block is an area with consistent text color in the text image to be recognized.

[0116] Text alignment refers to the order in which characters are arranged in the image to be recognized. It is typically horizontal (e.g., left to right or right to left) or vertical (e.g., top to bottom or bottom to top). In the image to be recognized, the alignment determines the reading order and spatial layout of the characters.

[0117] A text color block refers to an area with consistent text color in the text image to be recognized. Each text color block contains a single text character or multiple text characters that are arranged continuously and have consistent colors. Figure 2 Take the image to be identified as an example, which includes 6 text blocks, namely: "How things develop", "Written in the order of, the cause is the hunting dog", "Attack the little sparrow", "The old sparrow passed by", "Risk yourself to save the child", "The result is".

[0118] The text color block segmentation boundary refers to the boundary line used to distinguish adjacent text color blocks, and is used to accurately divide the range of the text color block. In the embodiment of the present application, the text color block segmentation boundary in the text image to be identified is distributed along the text arrangement direction. For example, in Figure 2 In the example text image to be recognized, the text characters are arranged horizontally. Therefore, the text color block segmentation boundaries are arranged horizontally in the text image to be recognized. A text color block segmentation boundary is a vertical line that is used to separate adjacent text characters with inconsistent colors. For another example, in the text image to be recognized, the text characters are arranged vertically. The text color block segmentation boundaries are arranged vertically in the text image to be recognized. A text color block segmentation boundary is a horizontal line that is used to separate adjacent text characters with inconsistent colors.

[0119] As for how to determine the text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized based on the above feature map, the following is Figure 8 The embodiments shown are only for explanation and will not be described in detail here.

[0120] In step 102, based on the feature map of the text image to be recognized, the text color block segmentation boundaries distributed along the text arrangement direction are determined in the text image to be recognized.

[0121] This step provides the basis for the subsequent text color recognition task.

[0122] Step 103: Divide the feature map into at least one feature sub-map according to the text color block segmentation boundary, wherein each feature sub-map corresponds to a text color block in the text image to be recognized.

[0123] In step 103, based on the text color block segmentation boundaries, the overall feature map generated in step 101 is divided into multiple independent feature sub-maps, each of which forms a one-to-one correspondence with a specific text color block. This means that each feature sub-map only contains the feature information of its corresponding text color block, thereby providing independent and exclusive feature input for the text color recognition task in the subsequent step 104. In this way, the interference of irrelevant information can be effectively reduced, allowing the subsequent text color recognition task to focus more on the color features of each independent text color block, thereby improving the efficiency and accuracy of text color recognition.

[0124] In one embodiment, the coordinates of the text block segmentation boundary are mapped to the spatial dimension of the feature map, and then the pixels or feature channels in the feature map are allocated to corresponding feature sub-maps based on the coordinates of the text block segmentation boundary to obtain the feature sub-map.

[0125] It should be noted that, when the colors of all text characters in the text image to be recognized are the same, a feature subgraph will be obtained, that is, the entire feature graph will be regarded as a feature subgraph. Figure 2 For the text image to be recognized in the example, multiple feature subgraphs will be obtained.

[0126] Step 104: Determine the color of the text in the corresponding text color block according to the characteristic sub-graph.

[0127] As can be seen from the description of step 103, a feature subgraph corresponds to a feature representation of a text color block. Therefore, in step 104, the color of the text in the corresponding text color block can be determined based on each feature subgraph. Here, the color of the text in the corresponding text color block can also be determined based on some of the feature subgraphs. For example, when identifying the color of text in some text color blocks in a text image, the text color can be identified based only on the feature subgraphs corresponding to the text color blocks to be identified.

[0128] In one embodiment, an exemplary implementation of determining the color of text in a corresponding text color block based on a feature subgraph includes: inputting the feature subgraph into a trained color regression prediction network to obtain the color of the text in the text color block corresponding to the feature subgraph.

[0129] See also Figure 3 , is a structural diagram of the color regression prediction network. Figure 3 As shown, the color regression prediction network includes a feature pooling layer, a highlight RGB regression module, and a text RGB regression module. Based on this, the feature sub-graph is input into the trained color regression prediction network to obtain the color of the text in the text color block corresponding to the feature sub-graph, including: inputting the feature sub-graph into the trained color regression prediction network, using the feature pooling layer to perform feature extraction processing on the feature sub-graph to obtain the feature vector of the corresponding text color block; using the highlight RGB regression module to extract the RGB regression prediction result of the highlight area in the corresponding text color block from the feature vector, and using the text RGB regression module to extract the RGB regression prediction result of the text area in the corresponding text color block from the feature vector; and determining the color of the text in the text color block based on the RGB regression prediction result of the highlight area in the text color block and the RGB regression prediction result of the text area in the text color block.

[0130] Specifically, the feature pooling layer is used for feature extraction processing on the input feature subgraph, generating feature vectors containing key information of the text color block. These feature vectors sufficiently retain the color information and semantic information of each text color block, laying a foundation for subsequent color regression prediction. Optionally, the feature pooling layer adopts global average pooling to eliminate information in the spatial dimension, obtaining the feature vector of the corresponding text color block. In this way, the computational complexity of subsequent color regression prediction can be reduced, providing a more concise and efficient solution for the final text color prediction.

[0131] The highlight RGB regression module and the text RGB regression module independently predict the three channels of color: R (red), G (green), and B (blue). The regression output value of each channel is in the normalized interval of 0 to 1. By multiplying this output value by 255, the color gray value of the corresponding channel can be directly obtained.

[0132] The highlight RGB regression module focuses on processing the highlight regions in the text color block, which usually contain key visual information. The highlight RGB regression module includes two fully connected layers. The fully connected layers are used to realize the full connection relationship between the input features and the output features, thereby learning and extracting the key information in the input features (i.e., the feature vectors mentioned above). The feature vectors first pass through the first fully connected layer for preliminary feature transformation and extraction, and then pass through the second fully connected layer to generate the RGB regression prediction results of the highlight regions.

[0133] The text RGB regression module focuses on processing the text information in the text color block. The text RGB regression module also includes two fully connected layers. The feature vectors first pass through the first fully connected layer for feature transformation, and then pass through the second fully connected layer to generate the RGB regression prediction results of the text regions. At this point, the highlight RGB regression module and the text RGB regression module generate their respective RGB prediction results, which can be fused in some way (such as weighted sum) to generate the final RGB regression prediction results.

[0134] The color regression prediction network shown in the above embodiment achieves RGB regression prediction of the highlight regions and text information in the text color block through the collaborative work of the highlight RGB regression module and the text RGB regression module. This structure enables the network to more comprehensively understand the image content and generate more accurate color prediction results. At the same time, this embodiment can directly regress the RGB three-channel gray value of the text color according to the feature vector, achieving an end-to-end text color prediction mode. Therefore, this method can effectively adapt to diverse text color features, ensuring robustness and accuracy in multi-color text scenarios.

[0135] The method provided by the embodiment of the present application, by extracting the feature map of the text image to be identified, determining the text color block segmentation boundary distributed along the text arrangement direction in the text image to be identified according to the feature map, realizes the local consistency feature of the text color directly parsed from the feature level, can more accurately divide the text color block boundary, and avoids the detection deviation caused by background interference or mask error in the traditional method; by dividing the feature map of the text image to be identified into at least one feature sub-graph according to the text color block segmentation boundary, and determining the color of the text in the corresponding text color block according to the feature sub-graph, realizes color prediction based on the color block as a whole, without the need to process each text character separately, so that when facing multi-color text, it can extract color information more efficiently, avoid the inefficiency of word-by-word processing, and ensure the accuracy of color prediction. In short, the method provided by the embodiment of the present application realizes end-to-end text color recognition, avoids dependence on upstream masks, and does not require additional post-processing steps, thereby significantly simplifying the calculation process, improving processing efficiency, and significantly improving the accuracy of text color recognition.

[0136] Figure 4 This is a flowchart of another text color recognition method provided in an embodiment of the present application. Figure 4 The process shown in Figure 1 Based on the process shown, the following steps are included:

[0137] Step 401: Generate a foreground mask image of the text image to be recognized.

[0138] The foreground mask image is used to identify the foreground region in the image, that is, the text area to be recognized. The foreground mask image helps to accurately extract and recognize the text content in subsequent processing steps.

[0139] In one embodiment, in order to generate a foreground mask image of a text image to be recognized, a foreground segmentation network based on U-Net is used. The network structure of U-Net is as follows: Figure 5 As shown in the figure, it adopts an encoder-decoder structure. The encoder is responsible for spatial information compression and feature extraction. Through a series of convolution and pooling operations, it gradually abstracts and compresses the features of the input image. The decoder, on the other hand, uses the encoder's intermediate layer features for upsampling, gradually restoring the image's spatial dimensions, and ultimately obtaining a foreground mask, also known as the foreground mask image.

[0140] Specifically, the structure of the foreground segmentation network is divided into two parts: the encoder (upsampling path) and the decoder (downsampling path). In the encoder, the input image first undergoes two 3×3 convolution operations, increasing the number of channels from 3 to 64, to obtain a preliminary feature map. Next, downsampling is performed through a 2×2 maximum pooling operation to reduce the spatial size of the feature map. This operation is repeated multiple times in the encoder, doubling the number of channels each time and further reducing the spatial size through maximum pooling, until the feature map of the last layer of the encoder is obtained.

[0141] For example, assuming the input image size is H×W×3 (where H and W are the image height and width, respectively, and 3 is the RGB color channel), the detailed operation process in the encoder part is as follows:

[0142] Phase 1: The input image is first convolved with two 3×3 convolution kernels. This step increases the number of image channels from the original 3 (representing the RGB color channels) to 64. Convolution extracts local features from the image and enriches them by increasing the number of channels. Therefore, after this step, a feature map of size H×W×64 is obtained.

[0143] After obtaining the H×W×64 feature map, a 2×2 max pooling operation is performed to reduce the spatial size (i.e., height and width) of the feature map while retaining important feature information. The max pooling operation selects the largest value within each 2×2 region as the representative of that region, thereby reducing the spatial size of the feature map while retaining the most important features. Therefore, after max pooling, the feature map size becomes H / 2×W / 2×64.

[0144] Phase 2:

[0145] Similar to the first stage, this stage also contains two 3×3 convolution operations, but this time the number of channels is doubled from 64 to 128. Through this operation, the features of the image are further extracted and enriched, resulting in a feature map of size H / 2×W / 2×128.

[0146] Similar to the first stage, a 2×2 max pooling operation is performed after the second stage to reduce the spatial size of the feature map. After this operation, the size of the feature map becomes H / 4×W / 4×128.

[0147] The above operation (i.e., two 3×3 convolutions followed by a 2×2 max pooling) is repeated twice in the encoder. In the third repetition, the number of channels doubles from 128 to 256, and the feature map size becomes H / 8×W / 8×256. In the fourth repetition, the number of channels doubles again to 512, and the feature map size becomes H / 16×W / 16×512.

[0148] In the final stage of the encoder, two 3x3 convolution operations are performed again to double the number of channels from 512 to 1024. After this step, a feature map with size H / 16xW / 16x1024 is obtained, which is the final feature map output by the encoder part.

[0149] Through a series of operations in the encoder part, the features of the image are gradually extracted and enriched, and the spatial size of the feature map is reduced, providing important feature information for the subsequent decoder part.

[0150] In the decoder part, each stage first doubles the spatial size through an upsampling (deconvolution) operation to gradually recover the spatial information of the image. Then, the upsampled feature map is concatenated with the feature map of the corresponding stage in the encoder in the channel dimension to fuse feature information at different levels. Next, two 3x3 convolution operations are performed to reduce the number of channels to further extract and abstract features. Such operations are repeated multiple times in the decoder part until the final foreground mask image with the same size as the input image is obtained.

[0151] For example, for a feature map of H / 16xW / 16x1024, the detailed operation process in the decoder part is as follows:

[0152] First, the H / 16xW / 16x1024 feature map is upsampled to H / 8xW / 8x1024. Upsampling is usually achieved by transposed convolution or bilinear interpolation, etc., which aims to increase the spatial size of the feature map.

[0153] Subsequently, the upsampled feature map is concatenated with the feature map of the corresponding stage in the encoder (size H / 8xW / 8x512) in the channel dimension. The concatenation operation is to stack two feature maps in the channel dimension to form a new feature map with size H / 8xW / 8x(1024+512) = H / 8xW / 8x1536.

[0154] The number of channels of the concatenated feature map is large, in order to reduce the computational complexity and avoid overfitting, two 3x3 convolution kernels are used for convolution operation to reduce the number of channels to 512. This step also helps to further extract and fuse features.

[0155] Next, the feature map obtained in the previous stage is upsampled to H / 4xW / 4 size and concatenated with the corresponding feature map in the encoder (size H / 4xW / 4x256) to form a feature map of H / 4xW / 4x(512+256) = H / 4xW / 4x768.

[0156] Again, adjust the channel number by two 3x3 convolution kernels to reduce the channel number to 256, obtaining a feature map with size H / 4xW / 4x256.

[0157] Continue upsampling and splicing: increase the size of the feature map to H / 2xW / 2, and splice it with the corresponding feature map in the encoder (with size H / 2xW / 2x128), obtaining a feature map with size H / 2xW / 2x(256+128)=H / 2xW / 2x384.

[0158] Finally, reduce the channel number to 128 by two 3x3 convolution kernels, obtaining a feature map with size H / 2xW / 2x128. Then, again, increase the size of the feature map to the size of the original image HxW by upsampling, and splice it with the initial feature map in the encoder (with size HxWx64), obtaining a feature map with size HxWx192. Then, adjust the channel number to the required output channel number by a 1x1 convolution kernel, here assuming 1 (i.e. single-channel output), obtaining the final output feature map with size HxWx1.

[0159] In summary, the decoder part gradually restores the spatial size of the feature map to the size of the original image by the process of gradually upsampling, splicing with the encoder feature map, and adjusting the channel number by convolution, and outputs the final prediction result.

[0160] In the foreground mask image, the prediction value of each pixel represents whether the current pixel has a foreground target (i.e. text content), and the prediction value ranges from 0 to 1. By setting a suitable threshold, the foreground mask image can be converted into a binary image, so as to accurately extract the text region. This method realizes accurate text segmentation, and provides strong support for subsequent character recognition and processing.

[0161] Step 402, input the text image to be recognized and its foreground mask image into the trained image feature extraction network, to obtain a feature map output by the image feature extraction network.

[0162] It can be seen from the description of step 402 that in an embodiment of the present application, the text image to be identified and its foreground mask image are jointly input into the trained image feature extraction network to obtain a feature map output by the image feature extraction network. For example, assume that the size of the text image to be identified is H×W×3, and the size of the foreground mask image is H×W×1. In the input stage, the foreground mask image is spliced ​​after the text image to be identified at the color channel level to form an input tensor of H×W×4. This processing method is intended to guide the neural network to adaptively focus on the text area based on the prior information of the foreground mask during the feature extraction stage, while implicitly suppressing the interference of background noise, thereby enhancing the robustness and fault tolerance of the model, and enabling the model to more comprehensively utilize background information, thereby showing higher accuracy and stability in text color recognition tasks.

[0163] Specifically, by combining the foreground mask as prior information with the original image (i.e., the image of the text to be recognized), the model learns during training how to combine these two inputs, adaptively focusing on the foreground region and implicitly removing background interference. Furthermore, compared to traditional methods that directly rely on the foreground mask to extract the foreground, this method reduces its reliance on the foreground mask. Even if the foreground mask contains certain errors (such as in scenarios with incomplete foreground regions or complex samples), the model can still perceive the text region through contextual information in the original image (such as color and texture), thereby reducing its reliance on the foreground mask quality and improving the system's robustness and fault tolerance. Furthermore, splicing the foreground mask as prior information into the original image and feeding it into the model effectively avoids directly discarding background pixels. This approach allows the model to explore the potential relationship between background and foreground during subsequent training, leveraging background information to assist in text color recognition. For example, in a text image with a complex background, the background color or texture may contrast with the text color, prompting the model to more accurately determine the true color of the text characters. This design enables the model to understand the image content more comprehensively and achieve higher accuracy in text color recognition tasks compared to traditional methods.

[0164] Step 403: Determine text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized based on the above feature map, wherein the text color block is an area with consistent text color in the text image to be recognized.

[0165] Step 404 : Divide the feature map into at least one feature sub-map according to the text color block segmentation boundary, wherein each feature sub-map corresponds to a text color block in the text image to be recognized.

[0166] Step 405: Determine the color of the text in the corresponding text color block according to the characteristic sub-graph.

[0167] The description of steps 403 to 405 can be found in the relevant description in the above embodiments, which will not be repeated here.

[0168] Figure 4 In the illustrated flow, by generating a foreground mask image of the to-be-identified text image, the to-be-identified text image and the foreground mask image thereof are jointly input into the trained image feature extraction network, the foreground mask is used as prior information to guide the model to adaptively focus on the foreground region, and the context information of the original image is combined to enable the model to implicitly reduce background interference and enhance the perception ability of the text region when extracting the feature map; even if the foreground mask has errors, the model can compensate for the missing information through the associated features of the original image, and through mining the potential relationship between the background and the foreground, the robustness and precision of the feature map to the text color representation are improved, providing a more reliable basis for subsequent text color block segmentation and color recognition.

[0169] In an embodiment, the image feature extraction network described above includes a feature encoding module and a feature pyramid module. The input end of the feature encoding module is the input end of the entire image feature extraction network, the output end of the feature encoding module is connected to the input end of the feature pyramid module, and the output end of the feature pyramid module is the output end of the entire image feature extraction network.

[0170] In this embodiment, the to-be-identified text image and the foreground mask image are input into the trained image feature extraction network to obtain the feature map output by the image feature extraction network, including: inputting the to-be-identified text image and the foreground mask image into the trained image feature extraction network, using the feature encoding module to perform multi-scale feature encoding processing on the to-be-identified text image and the foreground mask image to obtain a plurality of feature maps of different scales; using the feature pyramid module to perform cross-scale feature fusion processing on the plurality of feature maps of different scales to obtain the feature map of the to-be-identified text image.

[0171] Specifically, the feature encoding module includes M sequentially connected feature extraction units, and the feature pyramid module includes M-1 sequentially connected upsampling units. Accordingly, the feature extraction units are used to perform multi-scale feature extraction processing on the text image to be identified and the foreground mask image, including: using the first feature extraction unit to perform feature encoding processing on the text image to be identified and the foreground mask image to obtain a feature map at the first scale; using the i-th feature extraction unit to perform feature encoding processing on the feature map at the i-1-th scale to obtain a feature map at the i-th scale, where i takes values ​​one by one within the integer range [2, N]. A feature pyramid module is used to perform cross-scale feature fusion processing on feature maps at multiple scales to obtain a feature map of the text image to be recognized, including: using the first upsampling unit to perform feature fusion processing on the feature map at the Mth scale and the feature map at the M-1th scale to obtain the first feature fusion result; using the jth upsampling unit to perform feature fusion processing on the j-1th feature fusion result and the feature map at the sth scale to obtain the jth feature fusion result, where j is an integer value in the range of [2, M-1] and s is an integer value in the range of [M-2, 1]; and determining the M-1th feature fusion result as the feature map of the text image to be recognized. M is a natural number greater than 2.

[0172] For example, Figure 6 As shown in the figure, the feature encoding module includes four sequentially connected feature extraction units, which are labeled as the first feature extraction unit, the second feature extraction unit, the third feature extraction unit, and the fourth feature extraction unit for ease of description. The feature pyramid module includes three sequentially connected upsampling units, which are labeled as the first upsampling unit, the second upsampling unit, and the third upsampling unit for ease of description.

[0173] Among them, the input end of the first feature extraction unit is the input end of the entire feature extraction unit, the output end of the first feature extraction unit is connected to the input end of the second feature extraction unit, and is connected to the output end of the third upsampling unit through a convolution layer; the output end of the second feature extraction unit is connected to the input end of the third feature extraction unit, and is connected to the output end of the second upsampling unit through a convolution layer; the output end of the third feature extraction unit is connected to the input end of the fourth feature extraction unit, and is connected to the output end of the third upsampling unit through a convolution layer; the output end of the fourth feature extraction unit is the output end of the entire feature extraction unit, and is connected to the input end of the third upsampling unit. For example, Figure 7 As shown, each feature extraction unit includes four 3×3 convolutional layers and one 1×1 convolutional layer for deep feature extraction of the input feature map. It should be noted that even for convolution kernels of the same size in different convolutional layers, the parameters in the convolution kernels may be different, and this embodiment of the application does not limit this.

[0174] As can be seen, the feature encoding module consists of multiple feature extraction units, each of which contains four sequentially connected 3×3 convolutional layers, and features are fused through a 1×1 convolutional layer. This design can effectively perform multi-level feature extraction and encoding on the input image.

[0175] Specifically, if Figure 7 As shown, let the text image to be recognized be [H, W, 3], the foreground mask image be [H, W, 1], and the feature dimensions after the four feature extraction units of the feature encoding module are [H / 4, W / 4, 64], [H / 8, W / 8, 128], [H / 16, W / 16, 256],

[0176] [H / 32, W / 32, 512]. Then perform 1×1 convolution and upsampling operations on it to perform channel attenuation and upsampling respectively. After processing in sequence, the output feature is finally obtained, and its dimension is

[0177] [H / 4,W / 4,64], thus ensuring the preservation of feature space information.

[0178] Figure 6 and Figure 7 The network structure used in the example compensates for the lack of spatial detail in deep features. Specifically, after passing through the four feature extraction units of the feature encoding module, although the deep features are more expressive in semantic expression, their spatial resolution is significantly reduced (from [H, W, 3] to [H / 32, W / 32, 512]), which poses certain challenges to subsequent tasks such as text region segmentation. By introducing the feature pyramid module to fuse the intermediate layer features, not only is the computational complexity reduced, but also some spatial information can be restored through upsampling operations, allowing the fine-grained features of the intermediate layer to be preserved. This lays the foundation for the subsequent determination of the accuracy of text color block segmentation boundaries and the accuracy of text color recognition from the text image to be recognized based on the feature map of the text image to be recognized.

[0179] Figure 8 This is a flowchart of another text color recognition method provided in an embodiment of the present application. Figure 8 The process shown in Figure 1 and Figure 4 Based on the process shown in FIG, an exemplary implementation of determining the text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized according to the feature map of the text image to be recognized is described. Figure 8 As shown, the following steps are included:

[0180] Step 801: Input the feature map of the text image to be identified into the trained text region segmentation network to obtain prediction parameters output by the text region segmentation network, wherein the prediction parameters include confidence scores of each pixel band distributed along the text arrangement direction in the text image to be identified corresponding to the text color block segmentation boundary, and each pixel band is composed of a plurality of pixels continuously arranged in a first direction, and the first direction is perpendicular to the text arrangement direction.

[0181] Step 802: Determine text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized based on the prediction parameters.

[0182] For ease of understanding, steps 801 and 802 are explained uniformly here:

[0183] In conjunction with the above description of text block segmentation boundaries: Text block segmentation boundaries refer to the boundary lines used to distinguish adjacent text blocks and are distributed along the text arrangement direction. Therefore, in the process of determining text block segmentation boundaries, multiple pixel strips are determined in the text image to be recognized along the text arrangement direction. Each pixel strip is composed of multiple pixels arranged continuously in a first direction (perpendicular to the text arrangement direction).

[0184] by Figure 2 Taking the text image to be recognized as an example, the text characters are arranged horizontally, so multiple vertical pixel strips can be determined along the horizontal direction. Assuming that the width of the text image to be recognized is N pixels, then theoretically N vertical pixel strips can be determined along the horizontal direction.

[0185] Then, for each pixel band, the text region segmentation network predicts a confidence score corresponding to the text color block segmentation boundary based on the feature map of the text image to be identified. The confidence score reflects the text region segmentation network's judgment on whether the pixel band is a text color block segmentation boundary. The higher the score, the more the text region segmentation network believes that the pixel band is a text color block segmentation boundary. Based on this, the text color block segmentation boundary can be determined from multiple pixel bands based on the confidence score predicted by the text region segmentation network for each pixel band in the text image to be identified.

[0186] In one embodiment, determining text block segmentation boundaries distributed along the text arrangement direction in a text image to be recognized based on prediction parameters specifically includes: determining a set of candidate pixel bands from each pixel band based on the prediction parameters, wherein the confidence scores corresponding to the pixel bands in the candidate pixel band set are greater than or equal to a set threshold; and determining text block segmentation boundaries from the set of candidate pixel bands. This means that pixel bands with higher confidence scores are selected as potential text block segmentation boundaries.

[0187] As an optional implementation, in order to avoid repeated detection of multiple text block segmentation boundaries in a local area, the following method is used to determine the text block segmentation boundary from the candidate pixel band set: iterative processing is performed according to the following steps until all pixel bands in the candidate pixel band set are marked with the first mark. The specific steps are as follows: in the candidate pixel band set, the pixel band with the highest corresponding confidence score is determined among the pixel bands that are not marked with the first mark; the pixel band with the highest corresponding confidence score is recorded, and all pixel bands in the neighborhood window thereof are marked with the first mark; after the iterative processing is completed, all recorded pixel bands are determined as text block segmentation boundaries.

[0188] For ease of understanding, the following will be illustrated by way of example in conjunction with Figure 9

[0189] Figure 9 In the figure, each box represents a pixel band, and the numerical value in the box is the confidence score corresponding to the pixel band. According to the above description, first, by setting a threshold value, for example, 0.5, the pixel bands with higher confidence scores are filtered out from all the pixel bands to form a candidate pixel band set, and these pixel bands are determined as potential text block segmentation boundaries. Figure 9 In the figure, the confidence score corresponding to the pixel band not belonging to the candidate pixel band set is marked as 0, and the confidence scores of the pixel bands belonging to the candidate pixel band set are retained.

[0190] Next, the pixel band with the highest corresponding confidence score is determined in the candidate pixel band set (which is marked as “maximum value” in the Figure 9 figure), and the pixel band is recorded, for example, the index “4” of the pixel band is recorded. At the same time, all pixel bands in the neighborhood window of the pixel band with the highest corresponding confidence score are marked with the first mark. The neighborhood window is a window with a set length centered on the pixel band. Figure 9 In the figure, the first mark is embodied by setting the confidence scores of the pixel bands in the neighborhood window (shown as the suppression window in the Figure 9 figure) to 0.

[0191] Then, in the current candidate pixel band set, the pixel band with the highest corresponding confidence score is again determined from the pixel bands that are not marked with the first mark (also marked as “maximum value”), and the pixel band is recorded, for example, the index “12” of the pixel band is recorded, and all pixel bands in the neighborhood window thereof are marked with the first mark (also embodied by setting the confidence scores of the pixel bands in the neighborhood window to 0). At this point, all pixel bands in the candidate pixel band set are marked with the first mark, and the iterative processing is completed.

[0192] ​Finally, after the iterative process is completed, all recorded pixel bands are determined as text color block segmentation boundaries. For example, Figure 9 The two pixel bands with indices 4 and 12 are respectively determined as the text block segmentation boundaries.

[0193] Through the above steps, repeated detection of multiple text color block segmentation boundaries in a local area is effectively avoided, thereby improving the accuracy of text color block recognition.

[0194] Figure 8 The process shown in FIG1 is to input the feature map of the text image to be recognized into the trained text region segmentation network to obtain the prediction parameters output by the text region segmentation network. According to the prediction parameters, the text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized are determined, thereby directly analyzing the local consistency features of the text color from the feature level, and can more accurately divide the text color block boundaries, avoiding the detection bias caused by background interference or mask errors in traditional methods, thereby significantly improving the efficiency and accuracy of subsequent text color recognition.

[0195] In one embodiment, the text region segmentation network includes a downsampling module and a classification prediction module. The input of the downsampling module is the input of the entire text region segmentation network, the output of the downsampling module is connected to the input of the classification prediction module, and the output of the classification prediction module is the output of the entire text region segmentation network.

[0196] In this embodiment, the feature map is input into the trained text region segmentation network, and the specific implementation of obtaining the prediction parameters output by the text region segmentation network includes: inputting the feature map into the trained text region segmentation network, using the downsampling module to downsample the feature map layer by layer to obtain the downsampling processing results; and using the classification prediction module to obtain the prediction parameters based on the downsampling processing results.

[0197] Layer-by-layer downsampling involves performing a series of continuous downsampling operations on feature maps. This not only significantly reduces redundant information but also forms more compact and efficient feature vectors by encoding spatial information into the channel dimension. These feature vectors, rich in contextual semantic information, provide a solid and accurate foundation for subsequent text segmentation and position classification. This effectively reduces the computational complexity of the model, improving processing efficiency and prediction accuracy.

[0198] As an optional implementation, the downsampling module consists of N sequentially connected convolutional layers, each of which performs a convolution operation in the direction of the text alignment of the text image to be recognized. By adopting a one-dimensional convolution strategy in the direction of the text alignment, the model can fully and deeply capture the boundary information within the text area, making the segmentation boundaries between text of different colors more clear and distinct. This convolution operation focused on the text alignment enables the model to accurately locate the boundary areas of each color block, providing strong support and basis for the subsequent feature segmentation process.

[0199] The text region segmentation network provided by the above embodiment first gradually compresses the feature height through the downsampling module and encodes the spatial information into the feature channel. For example, by stacking three layers of downsampling modules, the height of the text image is reduced from the original H / 4 to H / 32, and finally the feature height is further removed using the global average pooling technology, retaining only the spatial information of the width dimension, and the final output feature dimension is [W / 4, C] (here is when the text is arranged horizontally). Based on these feature vectors, the classification prediction module of the segmentation position processes the features in the text arrangement direction through a one-dimensional convolution operation to predict the dividing points of different text color blocks. The segmentation position prediction result will be used to guide the block extraction of the feature, thereby decomposing the image into multiple independent regions, each of which corresponds to a single text color block, realizing color prediction based on the color block as a whole, without the need to process each text character separately, so that when facing multi-color text, color information can be extracted more efficiently, avoiding the inefficiency of word-by-word processing, and ensuring the accuracy of color prediction.

[0200] Figure 10 This is a schematic diagram of the overall structure of the text color recognition model provided in the embodiment of this application. Figure 10 As shown in the figure, the text color recognition model includes: foreground extraction network, feature extraction network, text region segmentation network (module), and color RGB regression network (module).

[0201] Among them, the foreground segmentation network is used to generate a foreground mask image of the text image to be recognized;

[0202] A feature extraction network is used to extract a feature map of the text image to be recognized based on the foreground mask image;

[0203] A text region segmentation network is used to determine the text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized based on the feature map of the text image to be recognized, wherein the text color block is the area with consistent text color in the image to be recognized;

[0204] The color RGB regression network is used to divide the feature map of the text image to be recognized into at least one feature sub-map based on the text color block segmentation boundary, where each feature sub-map corresponds to a text color block in the text image to be recognized; based on the feature sub-map, the color of the text in the corresponding text color block is determined.

[0205] Figure 10 The text color recognition model used as an example provides an end-to-end text color recognition method. In this method, the local consistency features of the text color are directly analyzed from the feature level, which can more accurately divide the boundaries of text color blocks and avoid the detection bias caused by background interference or mask errors in traditional methods. It also realizes color prediction based on the entire color block, without the need to process each text character separately. Therefore, when faced with multi-color text, it can more efficiently extract color information, avoid the inefficiency of word-by-word processing, and ensure the accuracy of color prediction.

[0206] As for Figure 10 The working details and interactions of each network (module) in the text color recognition model shown can be found in the description of the above embodiment and will not be repeated here.

[0207] Figure 11 This is a block diagram of an embodiment of a text color recognition device provided in an embodiment of the present application, such as Figure 11 As shown, the device includes:

[0208] Feature extraction module 1101, used to extract feature maps of the text image to be recognized;

[0209] a segmentation boundary determination module 1102 for determining, based on the feature map, text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized, wherein the text color block is an area with consistent text color in the text image to be recognized;

[0210] a segmentation module 1103, configured to divide the feature map into at least one feature sub-map according to the text color block segmentation boundary, wherein each feature sub-map corresponds to one of the text color blocks in the text image to be recognized;

[0211] The color recognition module 1104 is configured to determine the color of the text in the corresponding text color block according to the characteristic sub-graph.

[0212] In a possible implementation, the segmentation boundary determination module 1101 includes:

[0213] a prediction unit, configured to input the feature map into a trained text region segmentation network to obtain prediction parameters output by the text region segmentation network, wherein the prediction parameters include confidence scores corresponding to text color block segmentation boundaries of each pixel band distributed along the text arrangement direction in the to-be-recognized text image, each pixel band consisting of a plurality of pixels continuously arranged in a first direction, the first direction being perpendicular to the text arrangement direction;

[0214] The boundary determination unit is used to determine the text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized based on the prediction parameters.

[0215] In a possible implementation manner, the boundary determination unit is specifically configured to:

[0216] Determining a set of candidate pixel bands from each of the pixel bands according to the prediction parameters, wherein the confidence scores corresponding to the pixel bands in the set of candidate pixel bands are greater than or equal to a set threshold;

[0217] The text color block segmentation boundary is determined from the candidate pixel band set.

[0218] In a possible implementation manner, the boundary determination unit is specifically configured to:

[0219] The iterative process is performed according to the following steps until all pixel bands in the candidate pixel band set are marked with the first identifier:

[0220] Determining, among the pixel bands not marked with the first identifier in the candidate pixel band set, the pixel band with the highest corresponding confidence score;

[0221] Record the corresponding pixel band with the highest confidence score, and mark all pixel bands within its neighborhood window with a first identifier;

[0222] After the iterative process is completed, all recorded pixel bands are determined as text color block segmentation boundaries.

[0223] In one possible implementation, the text region segmentation network includes: a downsampling module and a classification prediction module; the prediction unit is specifically configured to:

[0224] Inputting the feature map into the trained text region segmentation network, and using the downsampling module to perform layer-by-layer downsampling processing on the feature map to obtain a downsampling processing result;

[0225] The classification prediction module is used to obtain the prediction parameters based on the downsampling processing result.

[0226] In a possible implementation, the downsampling module includes N sequentially connected convolutional layers, and each of the convolutional layers performs a convolution operation in a text arrangement direction of the text image to be recognized.

[0227] In a possible implementation, the feature extraction module includes:

[0228] a foreground mask generating unit, configured to generate a foreground mask image of the text image to be recognized;

[0229] The extraction unit is used to input the text image to be recognized and the foreground mask image into a trained image feature extraction network to obtain a feature map output by the image feature extraction network.

[0230] In one possible implementation, the image feature extraction network includes: a feature encoding module and a feature pyramid module; the extraction unit includes:

[0231] a multi-scale extraction subunit, configured to input the to-be-recognized text image and the foreground mask image into a trained image feature extraction network, and perform multi-scale feature encoding processing on the to-be-recognized text image and the foreground mask image using the feature encoding module to obtain a plurality of feature maps of different scales;

[0232] The fusion subunit is used to use the feature pyramid module to perform cross-scale feature fusion processing on the feature maps of the multiple different scales to obtain the feature map of the text image to be recognized.

[0233] In a possible implementation, the feature encoding module includes M feature extraction units connected in sequence, where M is a natural number greater than 2;

[0234] The multi-scale extraction subunit is specifically used to:

[0235] Using the first feature extraction unit to perform feature extraction processing on the text image to be recognized and the foreground mask image to obtain a feature map at a first scale;

[0236] The i-th feature extraction unit is used to perform feature extraction processing on the feature map at the i-1-th scale to obtain the feature map at the i-th scale, wherein the i is taken one by one in the integer range of [2, N].

[0237] In one possible implementation, the feature pyramid module includes M-1 sequentially connected upsampling units, and the fusion subunit is specifically configured to:

[0238] The first downsampling unit is used to perform feature fusion processing on the feature map at the Mth scale and the feature map at the M-1th scale to obtain the first feature fusion result;

[0239] Using the j-th downsampling unit to perform feature fusion processing on the j-1-th feature fusion result and the feature map at the s-th scale, to obtain the j-th feature fusion result, wherein the j is taken one by one in the integer range of [2, M-1], and the s is taken one by one in the integer range of [M-2, 1];

[0240] The M-1th feature fusion result is determined as the feature map of the text image to be recognized.

[0241] In a possible implementation, the color recognition module is specifically configured to:

[0242] Each of the feature sub-graphs is input into a trained color regression prediction network to obtain the color of the text in the text color block corresponding to each of the feature sub-graphs.

[0243] In one possible implementation, the color regression prediction network includes: a feature pooling layer, a highlight RGB regression module, and a text RGB regression module;

[0244] Inputting the feature subgraph into a trained color regression prediction network to obtain the color of the text in the text color block corresponding to the feature subgraph includes:

[0245] Inputting the feature subgraph into the trained color regression prediction network, performing feature extraction processing on the feature subgraph using the feature pooling layer to obtain a feature vector of the corresponding text color block;

[0246] Extracting the RGB regression prediction result of the highlight area in the corresponding text color block from the feature vector using the highlight RGB regression module, and extracting the RGB regression prediction result of the text area in the corresponding text color block from the feature vector using the text RGB regression module;

[0247] The color of the text in the text color block is determined according to the RGB regression prediction result of the highlight area in the text color block and the RGB regression prediction result of the text area.

[0248] like Figure 12 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0249] Memory 113, for storing computer programs;

[0250] In one embodiment of the present application, the processor 111 is configured to execute a program stored in the memory 113 to implement the text color recognition method provided by any of the aforementioned method embodiments, including:

[0251] Extract feature maps of text images to be recognized;

[0252] Determining, based on the feature map, text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized, wherein the text color block is an area with consistent text color in the text image to be recognized;

[0253] Dividing the feature map into at least one feature sub-map according to the text color block segmentation boundary, wherein each feature sub-map corresponds to one of the text color blocks in the text image to be recognized;

[0254] The color of the text in the corresponding text color block is determined according to the characteristic sub-graph.

[0255] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the text color recognition method provided in any of the aforementioned method embodiments are implemented.

[0256] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0257] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, or of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiment.

[0258] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "comprise", "include", "contain" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the specific order described or illustrated, unless the order of execution is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0259] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A text color recognition method, characterized in that: The method comprises: Extract feature maps of text images to be recognized; Determining, based on the feature map, text color block segmentation boundaries distributed along the text arrangement direction in the text image to be recognized, wherein the text color block is an area with consistent text color in the text image to be recognized; Dividing the feature map into at least one feature sub-map according to the text color block segmentation boundary, wherein each feature sub-map corresponds to one of the text color blocks in the text image to be recognized; The color of the text in the corresponding text color block is determined according to the characteristic sub-graph.

2. The method according to claim 1, characterized in that Determining text color block segmentation boundaries distributed along the text arrangement direction in the to-be-recognized text image according to the feature map includes: Inputting the feature map into a trained text region segmentation network to obtain prediction parameters output by the text region segmentation network, wherein the prediction parameters include confidence scores corresponding to text color block segmentation boundaries of each pixel band distributed along the text arrangement direction in the text image to be recognized, each pixel band consisting of a plurality of pixels continuously arranged in a first direction, the first direction being perpendicular to the text arrangement direction; According to the prediction parameters, text color block segmentation boundaries distributed along the text arrangement direction are determined in the text image to be recognized.

3. The method according to claim 2, characterized in that The step of determining text color block segmentation boundaries distributed along a text arrangement direction in the to-be-recognized text image according to the prediction parameters includes: Determining a set of candidate pixel bands from each of the pixel bands according to the prediction parameters, wherein the confidence scores corresponding to the pixel bands in the set of candidate pixel bands are greater than or equal to a set threshold; The text color block segmentation boundary is determined from the candidate pixel band set.

4. The method according to claim 3, characterized in that Determining the text color block segmentation boundary from the candidate pixel band set includes: The iterative process is performed according to the following steps until all pixel bands in the candidate pixel band set are marked with the first identifier: Determining, among the pixel bands not marked with the first identifier in the candidate pixel band set, the pixel band with the highest corresponding confidence score; Record the corresponding pixel band with the highest confidence score, and mark all pixel bands within its neighborhood window with a first identifier; After the iterative process is completed, all recorded pixel bands are determined as text color block segmentation boundaries.

5. The method according to claim 2, characterized in that The text region segmentation network includes: a downsampling module and a classification prediction module; the inputting the feature map into the trained text region segmentation network to obtain the prediction parameters output by the text region segmentation network includes: Inputting the feature map into the trained text region segmentation network, and using the downsampling module to perform layer-by-layer downsampling processing on the feature map to obtain a downsampling processing result; The classification prediction module is used to obtain the prediction parameters based on the downsampling processing result.

6. The method according to claim 5, characterized in that The downsampling module includes N sequentially connected convolution layers, and each of the convolution layers performs a convolution operation in the text arrangement direction of the text image to be recognized.

7. The method according to claim 1, characterized in that The step of extracting a feature map of a text image to be recognized includes: generating a foreground mask image of the text image to be recognized; The text image to be recognized and the foreground mask image are input into a trained image feature extraction network to obtain a feature map output by the image feature extraction network.

8. The method according to claim 7, characterized in that The image feature extraction network includes: a feature encoding module and a feature pyramid module; the text image to be recognized and the foreground mask image are input into the trained image feature extraction network to obtain a feature map output by the image feature extraction network, including: Inputting the to-be-recognized text image and the foreground mask image into a trained image feature extraction network, and performing multi-scale feature encoding processing on the to-be-recognized text image and the foreground mask image using the feature encoding module to obtain a plurality of feature maps of different scales; The feature pyramid module is used to perform cross-scale feature fusion processing on the feature maps of the multiple different scales to obtain the feature map of the text image to be recognized.

9. The method according to claim 8, characterized in that The feature encoding module includes M feature extraction units connected in sequence, where M is a natural number greater than 2; The step of performing multi-scale feature extraction processing on the to-be-recognized text image and the foreground mask image by using the feature encoding module includes: Using the first feature extraction unit to perform feature extraction processing on the text image to be recognized and the foreground mask image to obtain a feature map at a first scale; The i-th feature extraction unit is used to perform feature extraction processing on the feature map at the i-1-th scale to obtain the feature map at the i-th scale, wherein the i is taken one by one in the integer range of [2,M].

10. The method according to claim 9, characterized in that The feature pyramid module includes M-1 sequentially connected upsampling units; The step of performing cross-scale feature fusion processing on the feature maps of the plurality of different scales by using the feature pyramid module to obtain the feature map of the text image to be recognized includes: The first downsampling unit is used to perform feature fusion processing on the feature map at the Mth scale and the feature map at the M-1th scale to obtain the first feature fusion result; Using the j-th downsampling unit to perform feature fusion processing on the j-1-th feature fusion result and the feature map at the s-th scale, to obtain the j-th feature fusion result, wherein the j is taken one by one in the integer range of [2, M-1], and the s is taken one by one in the integer range of [M-2, 1]; The M-1th feature fusion result is determined as the feature map of the text image to be recognized.

11. The method according to claim 1, wherein Determining the color of the text in the corresponding text color block according to the characteristic subgraph includes: The feature sub-graph is input into a trained color regression prediction network to obtain the color of the text in the text color block corresponding to the feature sub-graph.

12. The method according to claim 11, characterized in that The color regression prediction network includes: a feature pooling layer, a highlight RGB regression module and a text RGB regression module; Inputting the feature subgraph into a trained color regression prediction network to obtain the color of the text in the text color block corresponding to the feature subgraph includes: Inputting the feature subgraph into the trained color regression prediction network, performing feature extraction processing on the feature subgraph using the feature pooling layer to obtain a feature vector of the corresponding text color block; Extracting the RGB regression prediction result of the highlight area in the corresponding text color block from the feature vector using the highlight RGB regression module, and extracting the RGB regression prediction result of the text area in the corresponding text color block from the feature vector using the text RGB regression module; The color of the text in the text color block is determined according to the RGB regression prediction result of the highlight area in the text color block and the RGB regression prediction result of the text area.

13. A text color recognition device, characterized in that: The device comprises: A feature extraction module is used to extract feature maps of the text image to be recognized; a segmentation boundary determination module, configured to determine, in the text image to be recognized, text color block segmentation boundaries distributed along a text arrangement direction based on the feature map, wherein the text color block is an area with consistent text color in the text image to be recognized; a segmentation module, configured to divide the feature map into at least one feature sub-map according to the text color block segmentation boundary, wherein each feature sub-map corresponds to one of the text color blocks in the text image to be recognized; The color recognition module is used to determine the color of the text in the corresponding text color block according to the characteristic sub-image.

Images