Banner text recognition method and related device

Abstract

The application discloses a banner text recognition method and related equipment. The method comprises the following steps: performing feature extraction based on a text region in a banner image to obtain a target feature map; obtaining a decoding feature map generated in the process of performing a decoding operation based on the target feature map, and performing skeleton prediction based on the target feature map to obtain a skeleton heat map; wherein the skeleton heat map represents the probability distribution of the existence of a text stroke skeleton at each pixel position, and the pixels in the decoding feature map represent the feature information of the corresponding pixel position in the banner image; using the skeleton heat map, performing gain on the pixels in the decoding feature map corresponding to the text stroke skeleton, and performing loss on the pixels in the decoding feature map corresponding to the non-text stroke skeleton, to obtain an enhanced feature map after the decoding feature map is guided by the skeleton heat map; and continuing to perform the decoding operation based on the enhanced feature map to obtain the recognized text of the text region. The above scheme can improve the accuracy of banner text recognition.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a banner text recognition method and related equipment. Background Technology

[0002] With the development of the field of visual processing, the recognition of text information in videos / images has become an important requirement.

[0003] Since hanging banners are present in many application scenarios, text recognition of the banners is required. In related technologies, OCR (Optical Character Recognition) technology is usually used to extract and recognize features from the original banner image.

[0004] However, due to factors such as fabric wrinkles, stretching deformation during hanging, partial occlusion, cluttered background interference, and font distortion, the image text may exhibit problems such as broken strokes and text deformation, resulting in low accuracy of banner text recognition. Summary of the Invention

[0005] The main technical problem addressed by this application is to provide a banner text recognition method and related equipment, which can improve the accuracy of banner text recognition.

[0006] The first aspect of this application provides a banner text recognition method, which includes: extracting features from a text region in a banner image to obtain a target feature map; acquiring a decoded feature map generated during the decoding operation based on the target feature map, and performing skeleton prediction based on the target feature map to obtain a skeleton heatmap; wherein the skeleton heatmap represents the probability distribution of the existence of text stroke skeletons at each pixel position, and the pixels in the decoded feature map represent the feature information of the corresponding pixel position in the banner image; using the skeleton heatmap, gaining the pixels in the decoded feature map corresponding to the text stroke skeleton, and subtracting the pixels in the decoded feature map corresponding to the non-text stroke skeleton, to obtain an enhanced feature map after the decoded feature map is guided by the skeleton heatmap; and continuing to perform decoding operations based on the enhanced feature map to obtain the recognized text of the text region.

[0007] The second aspect of this application provides a computer device including a memory and a processor coupled to each other, the memory storing program instructions, and the processor executing the program instructions to implement the banner text recognition method of the first aspect described above.

[0008] A third aspect of this application provides a computer-readable storage medium storing program instructions that, when executed by a processor, implement the banner text recognition method described in the first aspect above.

[0009] The fourth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the banner text recognition method described in the first aspect above.

[0010] The above scheme, based on the target feature map obtained after feature extraction of the text region in the banner image, produces a decoded feature map and a skeleton heatmap. The skeleton heatmap is then used to guide the decoded feature map, resulting in an enhanced feature map with more regular stroke structure and more accurate semantic representation. This enhanced feature map is then used for text recognition, reducing the problems of interference from complex backgrounds and distortion caused by directly using original image features. It effectively restores the true stroke structure and texture features of the banner text, significantly reducing the impact of invalid interference features on recognition, thereby improving the accuracy and robustness of text recognition. Therefore, it can improve the accuracy of banner text recognition.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in this application, the accompanying drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating an embodiment of the banner text recognition method of this application; Figure 2 This is a schematic diagram of the structure of an embodiment of the banner text recognition model of this application; Figure 3 This application Figure 1 A flowchart illustrating an embodiment of step S12; Figure 4 This application Figure 1 A flowchart illustrating an embodiment of step S13; Figure 5 This is a flowchart illustrating an embodiment of the training method for the banner text recognition model of this application; Figure 6 This application Figure 5 A flowchart illustrating an embodiment of step S21; Figure 7 This is a schematic diagram of the structure of an embodiment of the banner text recognition device of this application; Figure 8 This is a schematic diagram of the structure of an embodiment of the computer device of this application; Figure 9This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0014] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0015] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0016] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0017] Through long-term research, the inventors of this application have discovered that the task of recognizing banner text suffers from at least one of the following problems, such as the text being easily bent and wrinkled, environmental interference increasing the difficulty of recognition, and limitations of text recognition models, resulting in low accuracy in banner text recognition. For example, since banners are usually made of flexible materials (such as cloth or plastic), they are easily wrinkled and twisted when hung due to environmental factors such as wind, gravity, or fixing methods (such as rope binding), leading to text deformation, stroke misalignment, and other damage to the original structural features. Therefore, the text being easily bent and wrinkled results in low accuracy in banner text recognition. For example, when capturing banner images, the banner is often partially obscured by background objects such as tree branches and power lines, causing broken strokes or causing the algorithm to mistakenly identify interference as text strokes. Environmental interference increases the difficulty of recognition, resulting in low accuracy in banner text recognition. For example, text recognition models perform well in recognizing text in complete, unobstructed banner images, but wrinkles or occlusions on the banner can cause broken or deformed strokes, and background distractions (such as tree branches) may be misidentified as strokes, leading to recognition errors and low accuracy in banner text recognition.

[0018] To address the aforementioned technical problems, this application provides a banner text recognition method and related equipment to improve the accuracy of banner text recognition.

[0019] This application provides the following embodiments, and each embodiment is described in detail below.

[0020] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the banner text recognition method of this application. The method may include the following steps: S11: Extract features from the text region in the banner image to obtain the target feature map.

[0021] In some embodiments, the banner image is an image containing at least one banner to be identified. The banner to be identified in the banner image includes horizontally suspended banners (horizontal banners) and vertically suspended banners (vertical banners). Specifically, horizontal banners are suspended horizontally with a width greater than their height, while vertical banners are suspended vertically with a height greater than their width. It should be noted that the actual hanging direction of the banner to be identified and the display direction of the banner to be identified in the banner image may have a certain degree of rotational offset or perspective distortion, but the banner text recognition method described in this application can still be applied.

[0022] In some embodiments, the banner text recognition method of this application can be implemented using a banner text recognition model, that is, the banner text recognition model executes the steps of this application.

[0023] A banner image can be acquired, containing text information about the banner to be identified. For example, the text information can be characters in at least one language (such as Chinese characters), letters, numbers, etc., and this application does not limit the text information. Optionally, an image of the target area can be acquired to obtain the banner image; or, a target video can be acquired from the target area, and then the banner image can be determined based on the image frames of the target video, such as detecting the banner from the target video and extracting the banner area from the image frames of the target video to obtain the banner image, etc. This application does not limit the source of the banner image.

[0024] In some implementations, the banner image can be detected to obtain text regions; then, features can be extracted from the text regions to obtain a target feature map. Specifically, the target feature map can preliminarily characterize the basic visual features of different locations within the text region.

[0025] Optionally, a curved text detection algorithm can be used to detect the text information region in the banner image, thus obtaining the text region. For example, the ROI (Region of Interest) region of the text line can be selected as the text region. Optionally, features can then be extracted from the text region to obtain the target feature map. In some application scenarios, the text lines in the text region can be arranged horizontally from left to right (or from right to left), or vertically from top to bottom (or from bottom to top). In some application scenarios, the banner image to be identified simultaneously contains several text line regions arranged horizontally and several text line regions arranged vertically, and the banner text recognition method described in this application can still be applied.

[0026] Optionally, prior to the above steps, the banner image (or text region) can be preprocessed to obtain a preprocessed banner image. This preprocessing may include normalization and other preprocessing methods, which are not limited in this application. Therefore, the preprocessed banner image can be input into a banner text recognition model for banner text recognition.

[0027] S12: Obtain the decoded feature map generated during the decoding operation based on the target feature map, and perform skeleton prediction based on the target feature map to obtain the skeleton heatmap.

[0028] Decoded feature maps and skeleton heatmaps can be extracted from the target feature map. The decoded feature map can be used for character prediction to obtain text recognition results, and the pixels in the decoded feature map represent the feature information of the corresponding pixel position in the banner image. For example, the decoded feature map can be high-level features, deep features, etc.; this application does not limit the decoded feature map. The skeleton heatmap can be used to represent the text skeleton structure of the banner text region to be recognized in the banner image, and the skeleton heatmap represents the probability distribution of the existence of text stroke skeletons at each pixel position. Furthermore, the decoding operation can be implemented by a decoder such as a Transformer-based decoder; the network structure of the decoder is not limited here. It should be noted that the essence of the decoding operation is to gradually reconstruct the concrete text from the abstract feature map. In this process, the output feature map of the decoder's network layer (e.g., the attention mechanism layer) can be selected as the decoded feature map, or the hidden layer feature map of the decoder can also be selected as the decoded feature map; the specific source of the decoded feature map is not limited here.

[0029] Optionally, the skeleton heatmap is a pixel-level skeleton heatmap, that is, the skeleton heatmap contains the value of each pixel. The value of each pixel indicates whether the pixel belongs to a stroke region or a background region, or indicates the predicted probability that the pixel belongs to a stroke region or a background region. Stroke regions represent pixel regions containing text information, and background regions represent pixel regions containing non-text information. Stroke regions can also be called text regions, and this application does not limit this.

[0030] Optionally, the skeleton heatmap of this application can refer to a stroke skeleton binary map or a stroke skeleton heatmap. For example, taking a stroke skeleton binary map as an example, the skeleton heatmap can contain the value of each pixel (i.e., binary), and the value of each pixel is in the range of 0 or 1. A value of 1 indicates that the pixel belongs to the stroke region, and a value of 0 indicates that the pixel belongs to the background region. For example, taking a stroke skeleton heatmap as an example, the skeleton heatmap can contain the value of each pixel (i.e., the predicted probability). If the skeleton heatmap contains the predicted probability that each pixel belongs to the stroke region, the value of each pixel (predicted probability) is in the range of 0 to 1 (or can be in the form of a percentage). The closer the value is to 1 (or 100%), the more likely the pixel is to belong to the stroke region, and the closer the value is to 0 (or 0%), the more likely the pixel is to belong to the background region.

[0031] Optionally, single-task feature extraction can be performed on the target feature map of the text region of the banner image separately, such as performing character recognition and stroke skeleton detection on the target feature map of the text region of the banner image separately to obtain a decoded feature map and a skeleton heatmap. Optionally, multi-task feature extraction (such as character recognition and stroke skeleton detection) can be performed on the target feature map of the text region of the banner image to obtain a decoded feature map and a skeleton heatmap.

[0032] In some implementations, please refer to Figure 2 The banner text recognition model includes a feature extraction network, a character recognition branch, a stroke skeleton branch, a feature guidance network, and a prediction output network. The following section explains each step of the banner text recognition method in conjunction with the banner text recognition model.

[0033] In some embodiments, please refer to Figure 3 This embodiment can further extend step S12 of the above embodiment. Processing the banner image to obtain a decoded feature map and a skeleton heatmap can include the following steps: S121: Extract features from the text region in the banner image to obtain the target feature map.

[0034] A feature extraction network is used to extract features from the text regions in the banner image, resulting in a target feature map. The target feature map represents the visual features of the banner image and can be denoted as Fshared. Optionally, the feature extraction network can be a shared feature extractor. For example, the feature extraction network can be a backbone network (such as ResNet-34), a convolutional neural network, a lightweight network (such as GhostNet), etc. This application does not limit the feature extraction network.

[0035] Then, the target feature map is input into the character recognition branch and the stroke skeleton branch respectively.

[0036] S122: Extract deep features from the target feature map to obtain the decoded feature map.

[0037] The character recognition branch can be used to extract deep features from the target feature map to obtain the decoded feature map, or the character recognition branch can be used to perform character recognition on the target feature map to obtain the decoded feature map.

[0038] In some implementations, LSTM (Long Short Term Memory) is used to extract deep features from the target feature map to obtain the decoded feature map.

[0039] In some implementations, decoding the feature map includes extracting a first attention feature from the target feature map; attention processing can be performed on the target feature map to obtain the first attention feature. Optionally, the attention processing can be spatial attention processing, self-attention processing, etc., and this application does not limit the attention processing.

[0040] Optionally, an attention-based decoder (Transformer Encoder) is used to perform attention processing on the target feature map to obtain the first attention feature. For example, the attention-based decoder has 6 layers. Optionally, positional encoding can also be performed on the target feature map before each level of attention processing; for example, log-space encoding can be used for positional encoding.

[0041] For example, attention processing of the target feature map can be performed using the following formula: .

[0042] in, The first attention feature (decoded feature map) is represented by Q(Query), which represents the query vector of the target feature map. The initial query vector is... Each subsequent step updates the following: K (Key) represents the key vector of the target feature map, with each pixel corresponding to one K; V (Value) represents the target feature map, which is the value vector of the target feature map corresponding to K. The scaling score can be calculated based on Q and K. K and V are determined based on the target feature map. The initial query vector is obtained based on learnable parameters, and the next query vector can be output after each processing step.

[0043] For example, the initial query vector This includes multi-dimensional learnable parameters; for example, the initial query vector includes one 512-dimensional learnable parameter. The initial query vector can be obtained using the following formula. The formula is as follows: .

[0044] The above formula represents an embedding table for a target feature map, which contains a 512-dimensional learnable parameter.

[0045] It is understood that the decoded feature map can also be obtained in other ways, and the present application does not restrict the method of obtaining the decoded feature map.

[0046] S123: Generate a skeleton heatmap using the target feature map.

[0047] Stroke skeleton branches can be used to process the target feature map to generate a skeleton heatmap, which can guide the character recognition task.

[0048] In some implementations, the target feature map can be upsampled at a preset level, such as a 4-level upsampling process, to obtain upsampled features. Then, the upsampled features are used to generate a skeleton heatmap. Optionally, the upsampled features can be convolved and activated to obtain the skeleton heatmap.

[0049] For example, the stroke skeleton branch includes a decoder and a skeleton prediction network. The decoder can be a 4-layer upsampling decoder based on UNet (U-shaped Network, a deep learning model for image segmentation), which performs pre-defined upsampling processing on the target feature map to obtain upsampled features. Then, the data is input into the skeleton prediction network. The skeleton prediction network contains convolutional networks (such as one-dimensional convolutions) and activation functions (such as sigmoid), which sequentially upsample the features. Convolution and activation processing are performed to obtain the skeleton heatmap. For example, a skeleton heatmap. It can be calculated using the following formula: .

[0050] in, Represents a convolutional network. This represents the activation function Sigmoid.

[0051] Continue reading Figure 1 Following step S12 above, the following steps are also included: S13: Using the skeleton heatmap, the pixels corresponding to the text stroke skeleton in the decoded feature map are amplified, and the pixels corresponding to the non-text stroke skeleton in the decoded feature map are subtracted, resulting in an enhanced feature map after the decoded feature map is guided by the skeleton heatmap.

[0052] A feature-guided network is used to gain or lose weight on each pixel corresponding to the decoded feature map using a skeleton heatmap, so that the skeleton heatmap guides the character recognition task and obtains an enhanced feature map.

[0053] In some embodiments, please refer to Figure 4 This embodiment can further extend step S13 of the above embodiment. Using a skeleton heatmap, an enhanced feature map is obtained after the decoded feature map is guided by the skeleton heatmap. This embodiment may include the following steps: S131: Generate a spatial mask map using the skeleton heatmap.

[0054] A spatial mask map can be generated from a skeleton heatmap. This spatial mask map is used to numerically enhance pixels corresponding to text stroke skeletons and numerically suppress pixels corresponding to non-text stroke skeletons, and it has the same resolution as the decoded feature map. In some applications, the spatial mask map has higher values ​​for pixels corresponding to text stroke skeletons (i.e., pixels in the text region) and lower values ​​for pixels corresponding to non-text stroke skeletons (i.e., pixels in the background region).

[0055] Optionally, the skeleton heatmap can be sampled first to obtain a first heatmap, where the first heatmap and the decoded feature map have the same resolution. Specifically, the sampling process can be either upsampling or downsampling. After obtaining the first heatmap, an activation function (such as sigmoid) can be used to normalize the pixel values ​​of the skeleton heatmap, normalizing the values ​​of each pixel to [0,1], to obtain a second heatmap. Then, the second heatmap is used to generate a spatial mask.

[0056] In some implementations, adjustment parameters can be used to control the values ​​of each pixel in the second heatmap to obtain a spatial mask for each pixel, thus generating a spatial mask image. The adjustment parameters include compensation parameters and / or amplification parameters. The pixel value represents the predicted probability that the pixel belongs to a stroke region; for example, the closer the pixel value is to 1, the more likely the pixel is to belong to a stroke region; the closer the pixel value is to 0, the more likely the pixel is to belong to the background region. Optionally, the compensation parameter can be a value greater than 0 to compensate for low-response areas and prevent them from reaching zero. The amplification parameter can be a multiple or exponent greater than 1 to amplify high-response areas. Optionally, the pixel mask of each pixel and the compensation parameter can be added together to obtain the compensation result, and then the amplification parameter can be used to process the compensation result to obtain the spatial mask of each pixel.

[0057] Optionally, a spatial mask map is obtained by exponentially amplifying each pixel value in the second heatmap.

[0058] Alternatively, the spatial mask M of each pixel can be calculated using the following formula: .

[0059] in, represents the value of a pixel; 'a' represents the compensation parameter, such as 0.5, which is to prevent the low response area from returning to zero; 'b' represents the amplification parameter, such as 2, where the square operation is used to exponentially amplify the high response area.

[0060] For example, the spatial mask for each pixel can be as follows for the stroke area and the background area: such as the pixels in the stroke area ( ≈1), thereby controlling its spatial mask. This can be enhanced by up to 274%; such as the pixels in the background area ( ≈0), thus affecting its modulated spatial mask. This can suppress the background area.

[0061] S132: Element-wise multiplication of the decoded feature map is performed based on the spatial mask map to obtain the enhanced feature map.

[0062] After determining the spatial mask of each pixel to obtain the spatial mask map, the spatial mask map can be used to adjust (e.g., weight) the decoding feature map to guide the character recognition task and obtain an enhanced feature map.

[0063] In some implementations, the decoded feature map may include the first attention features of each pixel. The second attention feature can be obtained by weighting the first attention feature using the spatial mask of each pixel. This enhanced feature map is used to guide the character recognition task. Optionally, the enhanced feature map can also be normalized to obtain a normalized enhanced feature map.

[0064] Alternatively, the enhanced feature map can be calculated using the following formula: .

[0065] in, This represents the second attention feature (enhanced feature map). This represents the first attention feature (decoded feature map). Represents a spatial mask image. For element-wise dot product, This indicates normalization processing.

[0066] The above method, by using a spatial mask image to weight and adjust the decoded feature map, can enhance the network's attention weight to stroke regions and reduce the attention weight to non-stroke regions. This results in an enhanced feature map that focuses more on stroke regions and suppresses background interference. Furthermore, by relying on the stroke structure, contour direction, and skeleton topology represented by the skeleton heatmap, it accurately corrects defects in the decoded feature map caused by banner scene wrinkles, deformations, occlusions, uneven lighting, and font stretching and distortion, resulting in enhanced feature maps with more regular structures and more accurate semantic representations. This effectively restores the true stroke structure and texture features of the banner text, significantly reducing the impact of invalid interference features on recognition, thereby improving the accuracy and robustness of subsequent text recognition.

[0067] S14: Continue the decoding operation based on the enhanced feature map to obtain the recognized text of the text region.

[0068] A predictive output network can be used to perform text recognition using enhanced feature maps, thus obtaining the text recognition results for the banner image.

[0069] In some implementations, character prediction can be performed directly using the enhanced feature map to obtain the character sequence prediction probability; based on the character sequence prediction probability, the text recognition result is determined. For example, the prediction output network can be a CTC (Connectionist Temporal Classification) decoder. The CTC decoder performs character prediction on the enhanced feature map to obtain the character sequence prediction probability, which can then be converted into text information and parsed to obtain the final text recognition result.

[0070] Optionally, the network layer following the output decoded feature map continues to process the enhanced feature map to obtain the recognized text.

[0071] In some implementations, the enhanced feature map and the decoded feature map can be fused to obtain a new decoded feature map. Then, the new decoded feature map is used to perform decoding operations for character prediction, obtaining the character sequence prediction probability. Based on the character sequence prediction probability, the text recognition result is determined. Fusion can be achieved through addition, concatenation, weighted fusion, etc., and this application does not limit the type of fusion. The above method, by fusing the enhanced feature map and the decoded feature map to obtain a new decoded feature map before performing character prediction, not only introduces a skeleton heatmap into the text recognition task but also preserves as many original features of the text region of the banner image as possible, thereby enabling more accurate text recognition.

[0072] In some implementations, text line region information or text region information of the banner to be identified in the banner image is output based on the text recognition result, so as to facilitate subsequent application processing, such as correcting or recognizing curved text. This application does not limit this.

[0073] The above scheme, based on the target feature map obtained after feature extraction of the text region in the banner image, produces a decoded feature map and a skeleton heatmap. The skeleton heatmap is then used to guide the decoded feature map, resulting in an enhanced feature map with more regular stroke structure and more accurate semantic representation. This enhanced feature map is then used for text recognition, reducing the problems of interference from complex backgrounds and distortion caused by directly using original image features. It effectively restores the true stroke structure and texture features of the banner text, significantly reducing the impact of invalid interference features on recognition, thereby improving the accuracy and robustness of text recognition. Therefore, it can improve the accuracy of banner text recognition.

[0074] In some implementations, the above-mentioned banner text recognition method can be implemented using a banner text recognition model. Before performing step S11, that is, before acquiring the banner image, the banner text recognition model can be trained based on the sample banner image.

[0075] Please see Figure 5 , Figure 5 This is a flowchart illustrating an embodiment of the training method for the banner text recognition model of this application. The method may include the following steps: S21: Obtain the sample banner image.

[0076] Sample banner images can be obtained, each labeled with a corresponding ground truth heatmap and annotation information. These images can be used as training data to train the banner text recognition model. The method for obtaining sample banner images can be referenced; this application does not restrict the method of obtaining sample banner images.

[0077] Optionally, the annotation information of the sample banner image includes the sample recognition text in the text region of the sample banner image.

[0078] In some implementations, the truth heatmap includes the truth value (e.g., a binary 1 or 0) for each pixel to indicate whether the pixel belongs to a stroke region or a background region.

[0079] In some embodiments, please refer to Figure 6 This embodiment can further extend step S21 of the above embodiment. Obtaining the true heatmap may include the following steps: S211: Based on the banner hanging direction in the sample banner image, generate an anisotropic distance map of the sample banner image, and extract the skeleton based on the sample banner image to obtain the initial skeleton map of the text strokes.

[0080] An anisotropic distance map of a sample banner image can be generated. In the anisotropic distance map, the pixels represent the anisotropic distance from the text pixels to the edge of the text region in the sample banner image. The anisotropic distance represents the edge distance calculated by assigning horizontal metric weights in the horizontal direction and vertical metric weights in the vertical direction, respectively. The ratio between the horizontal metric weights and the vertical metric weights is related to the banner hanging direction.

[0081] In some implementations, the hanging direction of the sample recognition banner in the sample banner image can be determined first, and preprocessing can be performed based on the hanging direction. Optionally, the hanging direction includes horizontal hanging and vertical hanging. Optionally, the hanging direction can be determined using the aspect ratio (length to width ratio) or width-to-height ratio (width to height ratio) of the sample recognition banner in the sample banner image. For example, in response to the hanging direction being vertical, the sample banner image can be rotated by a preset angle (e.g., 90 degrees), and the sample banner image can be marked as rotated; in response to the hanging direction being horizontal hanging, no preprocessing is performed.

[0082] In some implementations, the text line mask can be determined first based on the sample banner image. For example, the sample banner image may be labeled with polygonal regions corresponding to text information and the corresponding text information. The polygonal regions of the sample banner image can be binarized to obtain the text line mask. Pixels located within the polygonal regions in the text line mask can have a value of 1, while pixels not located within the polygonal regions have a value of 0. Then, the anisotropic distance is determined for each pixel of the text information in the text line mask. Optionally, the text line mask can be used to determine each pixel in the sample banner image that belongs to the text information (e.g., a value of 1), thereby determining the anisotropic distance based on the coordinates of each pixel of the text information and the hanging direction of the sample banner image.

[0083] In some implementations, the anisotropic distance represents the weighted distance from a pixel to an edge. The anisotropic distance is obtained by weighting the corresponding directional distances using horizontal and vertical metric weights, respectively. The horizontal and vertical metric weights are related to the hanging direction of the sample recognition banner in the sample banner image. The directional distance of a pixel is calculated based on the coordinates of the pixel (such as the horizontal axis coordinates and the vertical axis coordinates).

[0084] In some implementations, when the sample recognition banner in the sample banner image is suspended horizontally, the horizontal metric weight is less than the vertical metric weight; or, when the sample recognition banner in the sample banner image is suspended vertically, the horizontal metric weight is not less than the vertical metric weight. In some application scenarios, the sum of the horizontal and vertical metric weights can be 1.

[0085] Optionally, based on the banner hanging direction in the sample banner image, determine the horizontal and vertical metric weights, and select a text pixel as the target pixel. Obtain the shortest horizontal edge distance of the target pixel from the edge of the text region in the horizontal direction, and obtain the shortest vertical edge distance of the target pixel from the edge of the text region in the vertical direction. In some application scenarios, there are several edge distances between the target pixel and the edge of the text region in the horizontal direction. Take the shortest one as the horizontal edge distance, and similarly obtain the shortest vertical edge distance.

[0086] Optionally, the horizontal edge distance is weighted using horizontal metric weights to obtain the horizontal anisotropic distance, and the vertical edge distance is weighted using vertical metric weights to obtain the vertical anisotropic distance. The edge distance of the target pixel is obtained based on the sum of the horizontal and vertical anisotropic distances. Anisotropic distance map of the sample strip image is generated using the edge distances of each target pixel.

[0087] Optionally, the edge distance between each pixel of the text information can be calculated using the following formula: .

[0088] in, Represents the edge distance of a pixel. Indicates the level of weight. This represents the vertical metric weight, where x and y represent the coordinates of the pixel (horizontal axis coordinate and vertical axis coordinate), respectively.

[0089] For example, when the sample banner image is suspended horizontally, the horizontal strokes droop and bend. In this case, setting the horizontal metric weight to be less than the vertical metric weight can strengthen the vertical weight and protect the continuity of the horizontal strokes. For example, the horizontal metric weight... The vertical metric weight is 0.3. The value is 0.7. For example, when the sample banner image is suspended vertically, the vertical strokes are distorted. In this case, setting the horizontal metric weight to be greater than the vertical metric weight can strengthen the horizontal weight and protect the structure of the vertical strokes.

[0090] S212: Based on the initial skeleton diagram and the anisotropic distance diagram, generate a truth heatmap of the text stroke skeleton.

[0091] An initial skeleton image can be generated using a sample banner image. Alternatively, an initial skeleton image can be generated using a text line mask from a sample banner image. The initial skeleton image can be obtained by skeleton extraction from either the sample banner image or the text line mask.

[0092] In some implementations, a preset skeleton extraction algorithm (such as the Zhang-Suen algorithm) can be used to extract the skeleton from the text line mask of the sample banner image, obtaining a single-pixel-wide skeleton image, i.e., a pixel-level initial skeleton map. The Zhang-Suen algorithm is a binary image thinning algorithm mainly used to extract the stroke skeleton of text or shapes from a text line mask. Its core principle is to iteratively remove boundary pixels while strictly maintaining the connectivity of the target, ultimately obtaining a single-pixel-wide skeleton image, i.e., the initial skeleton map. It is understood that other binary image thinning algorithms can also be used for skeleton extraction in this application; this application does not limit the preset skeleton extraction algorithm.

[0093] In some implementations, an initial skeleton map is generated from the sample banner image. Optionally, the initial skeleton map is generated using the text line mask of the sample banner image. For example, a preset skeleton extraction algorithm (such as the Zhang-Suen algorithm) can be used to extract the skeleton from the text line mask of the sample banner image to obtain the initial skeleton map. Then, connected components in the initial skeleton map are obtained, where each connected component corresponds to a line segment of a stroke, and the length of each stroke is the number of pixels contained in the connected component. Based on each connected component in the initial skeleton map, each stroke segment in the initial skeleton map is obtained, and a stroke length threshold is determined based on the aspect ratio of the text region in the sample banner image. Then, stroke segments in the initial skeleton map whose length is not greater than the stroke length threshold are filtered to obtain a new initial skeleton map. Since long banners are prone to having many skeleton breakpoints due to perspective shrinkage, fabric wrinkles, wind swaying, etc., this application determines the filtering threshold based on the character unit length of the sample banner image (or text line mask), which can adaptively and dynamically determine the filtering threshold to obtain a more accurate initial skeleton map.

[0094] Optionally, the character unit length can be determined based on the aspect ratio of the sample banner image. For example, the aspect ratio (the ratio of width to height) of the sample banner image can be used to characterize the character unit length of the text information. For instance, the character unit length R of the sample banner image is calculated as width / character height (or the width / height of the text line mask), which can then be represented as the minimum character length.

[0095] Optionally, the threshold range to which the character unit length R of the sample banner image (or text line mask) belongs can be determined, and a filtering threshold can be determined based on the threshold range. For example, in response to the character unit length R being less than or equal to the range threshold, a first threshold is determined as the filtering threshold; or, in response to the character unit length R being greater than the range threshold, a filtering threshold is determined based on a first threshold and a second threshold, and the maximum value of the first threshold and the second threshold can be selected as the filtering threshold; optionally, the second threshold is a preset multiple of the character unit length R. Optionally, the preset multiple is a multiple less than 1 or 0.5.

[0096] For example, the filtering threshold can be calculated using the following formula: .

[0097] Where S represents the range threshold, such as S = 8; R represents the character unit length; T1 represents the first threshold, such as T1 = 5; cR represents the second threshold, where c is a preset multiple, such as 0.15 times.

[0098] It is understood that the above are merely examples of the values, and the values ​​can be set to other values ​​as well, which is not limited in this application.

[0099] Therefore, by using a dynamic filtering threshold, connected components with stroke lengths less than the filtering threshold can be deleted from the initial skeleton graph to obtain the initial skeleton graph. This can be used to filter noise segments, such as filtering fragments smaller than 5px caused by wrinkles, as shown in the example above.

[0100] Therefore, the anisotropic distance is used to weight the new initial skeleton map. The anisotropic distance between the new initial skeleton map and each pixel position is multiplied pixel by pixel. This allows the corresponding distance weights to be retained for the skeleton positions (text information positions or stroke areas) in the sample skeleton map, while the pixel positions that are not in the initial skeleton map, i.e., non-skeleton positions (background areas), are set to zero, thus obtaining the initial skeleton heatmap.

[0101] In some implementations, the initial skeleton heatmap described above can be normalized to obtain a normalized ground truth heatmap, scaling the value of each pixel to the [0,1] range. For example, the ground truth value of each pixel in the initial skeleton heatmap can be normalized to the ratio of the pixel's value to its maximum value. For example, the normalized ground truth heatmap can be calculated using the following formula: .

[0102] in, A heatmap representing the normalized truth values. This represents the value of each pixel in the initial skeleton heatmap. The maximum value of each pixel in the initial skeleton heatmap.

[0103] Optionally, the values ​​of each pixel position in the initial skeleton heatmap are binarized to generate a ground truth heatmap of the text stroke skeleton.

[0104] The above method generates a ground truth heatmap of the text region of the sample banner in the sample banner image. It can calculate the anisotropic distance to address the problem of directional deformation of the banner caused by gravity. In addition, it introduces a dynamic length filtering threshold algorithm to filter out false broken strokes in the initial skeleton image and retain valid strokes, thereby obtaining a more accurate ground truth heatmap.

[0105] Continue reading Figure 5 Following step S21 above, the following steps are also included: S22: Train the banner text recognition model based on sample banner images, their annotation information, and ground truth heatmaps.

[0106] In some embodiments, a banner text recognition model is used to process sample banner images to obtain a predicted heatmap and predicted recognized text.

[0107] The predicted and recognized text can be target processing features, character sequence prediction probability, or text recognition result. This application uses character sequence prediction probability as an example for illustration.

[0108] The specific implementation process of this step can be referred to the description of the above embodiments, and will not be repeated here.

[0109] S23: Based on the predicted heatmap, the true heatmap, and the connected component weights, the first loss is obtained, and based on the predicted recognized text and the sample recognized text, the second loss is obtained.

[0110] The total loss of the banner text recognition model can be obtained based on the first loss and the second loss. The connected component weight is obtained based on the ratio between the number of target pixels in the longest connected component in the ground truth heatmap and the length of the target stroke in the connected component to which the pixel belongs.

[0111] In some implementations, positive sample loss terms and negative sample loss terms are determined using a predicted heatmap and a ground truth heatmap, respectively. The positive sample loss term represents the loss value of a pixel when it belongs to a stroke skeleton region in the ground truth heatmap, and the negative sample loss term represents the loss value of a pixel when it belongs to a non-stroke skeleton region in the ground truth heatmap. The sample loss terms are weighted using connected component weights to obtain a first loss. The sample loss term is the sum of the positive sample loss term and the negative sample loss term.

[0112] For example, the first loss can be obtained using the following formula: .

[0113] in, The first loss is represented by N; N represents the total number of pixels included, such as H×W (height×width), which is used to balance the loss magnitude of images at different resolutions. This represents a true heatmap containing the true values ​​of each pixel. The value range can be 0 or 1. It is used to mark the position of pixels in the stroke area. For example, if the value of pixel ij is 1, it means that the pixel belongs to the stroke area. If the value of pixel ij is 0, it means that the pixel belongs to the background area. This represents a predicted heatmap, containing the predicted value of each pixel. For example, the value range can be [0,1], such as the predicted probability that pixel ij belongs to a stroke region. This represents the positive sample loss term in the truth heatmap. Used when =1, to penalize missed detections (such as when a pixel is predicted as a stroke area but the true value is a background area); This represents the negative sample loss term in the truth heatmap. Used when =0, to penalize false detections (such as when a pixel is predicted as a background area but the true value is a stroke area).

[0114] The above approach treats all pixels equally, which may lead to occlusion noise competing with key strokes for optimization resources. Furthermore, short strokes and long strokes have the same gradient strength, causing the model to tend to learn simple short strokes (fast convergence but distorted structure). To address this issue, this application can also introduce connected component weights for loss weighting to obtain the first loss.

[0115] In some implementations, the first loss is obtained based on the ground truth heatmap, the predicted heatmap, and the connected component weights. The connected component weights of pixels in the ground truth heatmap are obtained based on the ratio between the number of target pixels in the longest connected component in the ground truth heatmap and the length of the target stroke in the connected component to which the pixel belongs. Positive and negative sample loss terms can be determined using the predicted and ground truth heatmaps, respectively. The positive sample loss term represents the loss value when a pixel belongs to a stroke region in the ground truth heatmap, and the negative sample loss term represents the loss value when a pixel belongs to a background region in the ground truth heatmap. The sample loss terms are then weighted using the connected component weights to obtain the first loss. The sample loss term is the sum of the positive and negative sample loss terms, and the weighted loss can be dynamically perceived through connected components.

[0116] For example, the first loss can be obtained using the following formula: .

[0117] in, The first loss is represented by N; N represents the total number of pixels included, such as H×W (height×width), which is used to balance the loss magnitude of images at different resolutions. This represents the weight of the connected components, which is used to dynamically adjust the loss weight based on the stroke length. This represents a true heatmap containing the true values ​​of each pixel. The value range can be 0 or 1. It is used to mark the position of pixels in the stroke area. For example, if the value of pixel ij is 1, it means that the pixel belongs to the stroke area. If the value of pixel ij is 0, it means that the pixel belongs to the background area. This represents a predicted heatmap, containing the predicted value of each pixel. For example, the value range can be [0,1], such as the predicted probability that pixel ij belongs to a stroke region. This represents the positive sample loss term in the truth heatmap. Used when =1, to penalize missed detections (such as when a pixel is predicted as a stroke area but the true value is a background area); This represents the negative sample loss term in the truth heatmap. Used when =0, to penalize false detections (such as when a pixel is predicted as a background area but the true value is a stroke area).

[0118] In some implementations, the connected component weights of each pixel point ij in the truth heatmap It can be obtained through the following formula: .

[0119] in, This represents the number of target pixels in the longest connected region in the truth heatmap, which is also the number of pixels contained in the longest stroke. This represents the target stroke length of the connected component to which pixel ij belongs. Optionally, the stroke length can be obtained based on the number of pixels contained in the stroke segment.

[0120] Optionally, the mask of the truth heatmap can be segmented into connected components in the neighborhood to determine the connected components (i.e., strokes) of each pixel in the truth heatmap. Each stroke contains several pixels, or it can be represented as a line segment, which can be a line segment composed of all pixels in each connected component. For example, the truth heatmap can be segmented into connected components by 8 neighborhoods to obtain the strokes of each connected component.

[0121] Then, obtain the stroke length (or number of pixels) corresponding to each connected component. And obtain the target pixel count Lmax of the longest connected component in the true heatmap, where each connected component corresponds to a stroke, the stroke length can be the number of pixels contained in the stroke, and the pixel count Lmax of the longest stroke is the maximum value of the stroke length.

[0122] Next, a pixel is selected as the target pixel, and the target stroke length of the connected component to which the target pixel belongs is obtained. Based on the ratio between the target stroke length and the number of target pixels, the connected component weights corresponding to each pixel are determined. Therefore, the connected component weights corresponding to each pixel can be determined. The mapping to the space matrix is ​​used in loss calculation.

[0123] Optionally, the square root of the ratio between the target stroke length and the target number of pixels is used as the connected component weights for calculating the first loss.

[0124] The above method, by obtaining dynamic connected component weights and using the connected component weights of stroke length to weight the stroke skeleton loss, can reduce the impact of broken strokes in banner text.

[0125] The second loss can be determined by calculating the loss based on the predicted character recognition result. Optionally, the second loss can be obtained by calculating the loss using the predicted character recognition result and the corresponding ground truth predicted character recognition result. The method of calculating the second loss in this application is not limited to this.

[0126] Combining the first loss and the second loss, we can obtain the total loss value.

[0127] In some implementations, the first loss is weighted using a weighting factor to obtain a weighted first loss, and then the weighted first loss and the second loss are combined (e.g., by adding them together) to obtain the total loss value. For example, the total loss value can be calculated using the following formula: .

[0128] in, This is the total loss value. As a weighting factor, This is the second loss. This is the first loss.

[0129] In some implementations, the weighting factor is related to the training period of the banner text recognition model. Optionally, the weighting factor is positively correlated with the training period of the banner text recognition model. Optionally, the weighting factor in the first period of the training period is less than the weighting factor in the second period, wherein the second period is later than the first period. This application does not impose any specific limitations on the weighting factor. For example, λ is 0.2 in the early stage of training and λ is 0.5 in the later stage of training, thereby avoiding the interference of inaccurate skeleton heatmap prediction in the early stage of training with the main task of text recognition (i.e., character recognition).

[0130] S24: A banner text recognition model trained based on the joint training of the first and second losses.

[0131] Based on the total loss value mentioned above, the parameters of the banner text recognition model are adjusted to obtain the adjusted banner text recognition model. Optionally, the above steps can be executed iteratively a preset number of times to obtain the final trained banner text recognition model.

[0132] The above scheme introduces both ground truth heatmaps and connected component weights from sample banner images, allowing the banner text recognition model to simultaneously output predicted heatmaps and predicted recognized text during training. First and second losses are calculated separately, and the total loss is obtained by combining them. Model parameters are then adjusted based on this total loss. On one hand, the first loss, related to stroke skeleton prediction, constrains the model to learn accurate text stroke topology and skeleton morphology, correcting stroke feature distortion caused by banner deformation and wrinkles. On the other hand, the second loss, related to character recognition, supervises character classification representation capabilities, achieving joint optimization of skeleton structure constraints and text recognition tasks. This results in a trained model with both robust stroke feature correction capabilities and high-precision text recognition capabilities, significantly improving the model's generalization and recognition robustness in tilted, deformed, and low-quality banner scenarios.

[0133] It is understood that in the above method of specific implementation, the order in which each step is written does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0134] In some embodiments, this application also provides a banner text recognition device for implementing the banner text recognition method of any of the above embodiments.

[0135] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an embodiment of the banner text recognition device of this application. The banner text recognition device 30 includes an extraction module 31, a prediction module 32, a guidance module 33, and a decoding module 34. The modules are interconnected.

[0136] The extraction module 31 is used to extract features from the text region in the banner image to obtain a target feature map. The prediction module 32 is used to obtain the decoded feature map generated during the decoding operation based on the target feature map, and to perform skeleton prediction based on the target feature map to obtain a skeleton heatmap; wherein, the skeleton heatmap represents the probability distribution of the existence of text stroke skeletons at each pixel position, and the pixels in the decoded feature map represent the feature information of the corresponding pixel position in the banner image. The guidance module 33 is used to use the skeleton heatmap to gain the pixels corresponding to text stroke skeletons in the decoded feature map, and to subtract the pixels corresponding to non-text stroke skeletons in the decoded feature map, resulting in an enhanced feature map after the decoded feature map has been guided by the skeleton heatmap. The decoding module 34 is used to continue the decoding operation based on the enhanced feature map to obtain the recognized text of the text region.

[0137] It should be noted that the banner text recognition device and the banner text recognition method provided in the above embodiments belong to the same concept. The specific operation methods of each module and unit have been described in detail in the method embodiments and will not be repeated here. In practical applications, the banner text recognition device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This application does not impose any limitations on this.

[0138] It is understood that the banner text recognition method and the banner text recognition model training method in this application can be executed by a computer device. This computer device can be any device with processing capabilities, such as a mobile device, computer, server, etc., and this application does not impose any restrictions on it. In some possible implementations, the above method can be implemented by the processor calling program instructions stored in memory.

[0139] In some embodiments, please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of this application. The computer device 40 includes a memory 41 and a processor 42 coupled to each other. The memory 41 stores program instructions, and the processor 42 executes the program instructions to implement the steps of any embodiment of the banner text recognition method and the banner text recognition model training method described above. In a specific implementation scenario, the computer device 40 may include, but is not limited to, a microcomputer and a server. In addition, the computer device 40 may also include mobile devices such as laptops and tablets, which are not limited here.

[0140] In this embodiment, processor 42 can also be referred to as a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 42 can be any conventional processor. Furthermore, processor 42 can be implemented using integrated circuit chips.

[0141] The methods described in the above embodiments can be implemented as computer programs; therefore, this application proposes a computer-readable storage medium. Please refer to [link to relevant documentation]. Figure 9 , Figure 9 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor. The program instructions 51 can be executed by the processor to implement the steps of any embodiment of the banner text recognition method and the banner text recognition model training method described above.

[0142] The computer-readable storage medium 50 in this embodiment includes: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program instructions 51. Alternatively, it may be a server storing the program instructions 51. The server can send the stored program instructions 51 to other devices for execution, or it can automatically execute the stored program instructions 51.

[0143] This application also provides a computer program product comprising a computer program that, when executed by a processor, can implement the steps of the methods described in any of the foregoing embodiments. Specifically, the computer program product can be a software or program product containing a computer program, capable of running on a computing device or stored on any available medium.

[0144] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments. For the sake of brevity, this application will not repeat the details here.

[0145] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to. For the sake of brevity, the present application will not repeat them here.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0147] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0148] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0149] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application.

[0150] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, and thus stored in a computer-readable storage medium for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, this application is not limited to any particular hardware and software combination.

[0151] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for recognizing banner text, characterized in that, include: Feature extraction is performed on the text region in the banner image to obtain the target feature map; A decoded feature map generated during the decoding operation based on the target feature map is obtained, and a skeleton prediction is performed based on the target feature map to obtain a skeleton heatmap; wherein, the skeleton heatmap represents the probability distribution of the existence of text stroke skeletons at each pixel position, and the pixels in the decoded feature map represent the feature information of the corresponding pixel position in the banner image; Using the skeleton heatmap, pixels corresponding to text stroke skeletons in the decoded feature map are amplified, and pixels corresponding to non-text stroke skeletons in the decoded feature map are subtracted, resulting in an enhanced feature map after the decoded feature map is guided by the skeleton heatmap. Based on the enhanced feature map, the decoding operation is continued to obtain the recognized text of the text region.

2. The method according to claim 1, characterized in that, The process of using the skeleton heatmap to gain the pixels corresponding to text stroke skeletons in the decoded feature map and to subtract the pixels corresponding to non-text stroke skeletons in the decoded feature map, resulting in the enhanced feature map guided by the skeleton heatmap, includes: Based on the skeleton heatmap, a spatial mask map is generated; wherein, the spatial mask map is used to perform numerical enhancement on pixels corresponding to text stroke skeletons and numerical suppression on pixels corresponding to non-text stroke skeletons, and the spatial mask map has the same resolution as the decoded feature map. The enhanced feature map is obtained by multiplying the decoded feature map element-wise based on the spatial mask map.

3. The method according to claim 2, characterized in that, The step of generating a spatial mask map based on the skeleton heatmap includes: Based on the skeleton heatmap, a first heatmap with the same resolution as the decoded feature map is obtained by sampling. The pixel values ​​of the first heatmap are normalized to obtain the second heatmap. The spatial mask image is obtained by exponentially amplifying each pixel value in the second heatmap.

4. The method according to claim 1, characterized in that, The step of continuing the decoding operation based on the enhanced feature map to obtain the recognized text of the text region includes: A new decoding feature map is obtained by fusing the enhanced feature map and the decoded feature map. Based on the new decoding feature map, the decoding operation is continued to obtain the recognized text.

5. The method according to any one of claims 1 to 4, characterized in that, The recognized text is generated based on a banner text recognition model, and the training steps of the banner text recognition model include: Based on the banner hanging direction in the sample banner image, an anisotropic distance map of the sample banner image is generated, and skeleton extraction is performed based on the sample banner image to obtain an initial skeleton map of text strokes; wherein, the pixels in the anisotropic distance map represent the anisotropic distance from the text pixel to the edge of the text region in the sample banner image, the anisotropic distance represents the edge distance calculated by assigning horizontal metric weights in the horizontal direction and vertical metric weights in the vertical direction, and the ratio between the horizontal metric weights and the vertical metric weights is related to the banner hanging direction; Based on the initial skeleton diagram and the anisotropic distance diagram, a truth heatmap of the text stroke skeleton is generated. The banner text recognition model is trained based on the sample banner images, their annotation information, and the ground truth heatmap; wherein the annotation information includes the sample recognition text in the sample banner images.

6. The method according to claim 5, characterized in that, The step of generating an anisotropic distance map of the sample banner image based on the banner hanging direction in the sample banner image includes: Based on the banner hanging direction in the sample banner image, the horizontal metric weight and the vertical metric weight are determined, and a text pixel is selected as the target pixel. The shortest horizontal edge distance from the target pixel to the edge of the text region in the horizontal direction is obtained, and the shortest vertical edge distance from the target pixel to the edge of the text region in the vertical direction is obtained. Wherein, in response to the hanging direction being horizontal, the horizontal metric weight is less than the vertical metric weight; in response to the hanging direction being vertical, the horizontal metric weight is not less than the vertical metric weight. The horizontal anisotropic distance is obtained by weighting the horizontal edge distance using the horizontal metric weight, and the vertical anisotropic distance is obtained by weighting the vertical edge distance using the vertical metric weight. The edge distance of the target pixel is obtained based on the sum of the horizontal anisotropic distance and the vertical anisotropic distance; An anisotropic distance map of the sample banner image is generated using the edge distances of each target pixel.

7. The method according to claim 5, characterized in that, The step of generating a ground truth heatmap of text stroke skeletons based on the initial skeleton map and the anisotropic distance map includes: Based on each connected component in the initial skeleton graph, each stroke segment in the initial skeleton graph is obtained, and based on the aspect ratio of the text region in the sample banner image, the stroke length threshold is determined. Filter the stroke segments in the initial skeleton diagram whose length is not greater than the stroke length threshold to obtain a new initial skeleton diagram; The initial skeleton map is obtained by multiplying the new initial skeleton map with the anisotropic distance of each pixel position pixel by pixel, and setting the pixel positions that are not in the initial skeleton map to zero. The values ​​of each pixel position in the initial skeleton map are binarized to generate the ground truth heatmap of the text stroke skeleton.

8. The method according to claim 5, characterized in that, The step of training the banner text recognition model based on the sample banner images and their annotation information and the ground truth heatmap includes: The sample banner image is processed using the banner text recognition model to obtain a predicted heatmap and predicted recognized text. Based on the predicted heatmap, the ground truth heatmap, and the connected component weights, a first loss is obtained, and based on the predicted recognition text and the sample recognition text, a second loss is obtained; wherein, the connected component weights are obtained based on the ratio between the number of target pixels in the longest connected component in the ground truth heatmap and the length of the target stroke in the connected component to which the pixel belongs; The banner text recognition model is jointly trained based on the first loss and the second loss.

9. The method according to claim 8, characterized in that, The first loss is obtained based on the predicted heatmap, the true heatmap, and the connected component weight matrix, including: Using the predicted heatmap and the ground truth heatmap, positive sample loss term and negative sample loss term are determined respectively; wherein, the positive sample loss term represents the loss value of a pixel when it belongs to the stroke skeleton region in the ground truth heatmap, and the negative sample loss term represents the loss value of a pixel when it belongs to the non-stroke skeleton region in the ground truth heatmap. The sample loss term is weighted using the connected component weights to obtain the first loss; wherein the sample loss term is the sum of the positive sample loss term and the negative sample loss term.

10. The method according to claim 8, characterized in that, The steps for obtaining the connected component weights include: The mask of the truth heatmap is segmented into neighboring connected components to determine the connected component to which each pixel in the truth heatmap belongs, and the number of target pixels in the longest connected component in the truth heatmap is obtained. Select a pixel as the target pixel, and obtain the target stroke length of the connected component to which the target pixel belongs; The square root of the ratio between the target stroke length and the target pixel count is used to map the square root result to a spatial matrix as the connected component weights for calculating the first loss.

11. A computer device, characterized in that, The method includes a memory and a processor coupled to each other, wherein the memory stores program instructions and the processor executes the program instructions to implement the banner text recognition method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The device stores program instructions that, when executed by a processor, implement the banner text recognition method according to any one of claims 1 to 10.

13. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the banner text recognition method according to any one of claims 1 to 10.

Images