Text recognition method and storage medium

Abstract

The invention discloses a text recognition method and a storage medium, and the method comprises the steps: carrying out the text segmentation of a to-be-recognized image, and obtaining a segmented image containing a text region; performing binary marking processing on the segmented image according to pixel information of the text region to obtain an image feature mask; performing sampling processing on to-be-recognized features of the to-be-recognized image according to the image feature mask to obtain sampling features; inputting the sampling features and the to-be-recognized features into a pre-trained attention network for feature extraction processing to obtain target image features; and determining a corresponding target text according to the target image feature. According to the scheme, the accuracy of text recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a text recognition method and a storage medium. Background Art

[0002] Optical Character Recognition (OCR) is used in a large number of scenarios in real life. However, due to differences in character styles, fonts, and the influence of various external environments, the text recognition model is prone to performance degradation or misrecognition. At present, the Large Language Model (LLM) technology based on the Transformer architecture has achieved rapid development. Its core principle is to use the self-attention mechanism to capture the dependencies in the input sequence, so as to handle sequence-to-sequence tasks such as machine translation, text generation, and question-answering systems. On this basis, the same Vision Transformer method is used in the visual field, with the help of its global attention mechanism, which greatly improves the versatility and robustness of various methods in the visual field, such as recognizing text information in images.

[0003] However, the computational complexity of the self-attention mechanism is positively correlated with the number of tokens (the basic unit of the input sequence). Too many tokens will not only make the model training process difficult, but also make the training and reasoning processes more time-consuming and resource-intensive, and reduce the efficiency of the attention mechanism. Summary of the invention

[0004] The present application at least provides a text recognition method, apparatus, device and computer-readable storage medium.

[0005] The first aspect of the present application provides a text recognition method, including: performing text segmentation processing on an image to be recognized to obtain a segmented image containing a text area; performing binary labeling processing on the segmented image according to pixel information of the text area to obtain an image feature mask; performing sampling processing on the features to be recognized of the image to be recognized according to the image feature mask to obtain sampling features; inputting the sampling features and the features to be recognized into a pre-trained attention network for feature extraction processing to obtain target image features; and determining the corresponding target text according to the target image features.

[0006] In one embodiment, the image feature mask includes a foreground area and a background area, and the sampling process of the features to be identified of the image to be identified based on the image feature mask to obtain the sampling features includes: determining the number of sampling times based on the number of the foreground areas; and performing discrete sampling process on the features to be identified of the image to be identified based on the sampling times to obtain the sampling features.

[0007] In one embodiment, before performing text segmentation processing on the image to be recognized to obtain a segmented image containing a text area, the method further includes: resizing the received original image to obtain an image to be processed that matches a preset size; performing segmentation and embedding processing on the image to be processed to obtain the image to be recognized and the features to be recognized of the image to be recognized.

[0008] In one embodiment, the text segmentation processing is performed on the image to be recognized to obtain a segmented image containing text areas, including: inputting the to-be-recognized features of the image to be recognized into a pre-trained image segmentation network to perform segmentation processing and font standardization processing to obtain the segmented image; the font types of each text area in the segmented image are the same.

[0009] In one embodiment, the step of inputting the features to be identified of the image to be identified into a pre-trained image segmentation network for segmentation processing and font normalization processing to obtain the segmented image includes: inputting the features to be identified of the image to be identified into a pre-trained image segmentation network for downsampling and feature extraction processing to obtain the downsampled features of the features to be identified; and upsampling the downsampled features through the image segmentation network to obtain a segmented image containing the text area.

[0010] In one embodiment, the binary labeling processing is performed on the segmented image according to the pixel information of the text area to obtain the image feature mask, including: partitioning the segmented image to obtain at least one image area; determining the binary label corresponding to each image area according to the pixel category of each pixel point in each image area to obtain the image feature mask.

[0011] In one embodiment, the segmented image includes the text area and the non-text area, the pixels in the text area are foreground pixels, and the pixels in the non-text area are background pixels, and the binary labels corresponding to each image area are determined according to whether each pixel in each image area is in the text area to obtain the image feature mask, including: obtaining the foreground quantity ratio of the foreground pixels and the background quantity ratio of the background pixels in each image area; performing a first labeling process on the image area where the foreground quantity ratio is greater than the background quantity ratio to obtain a first labeling value; performing a second labeling process on the image area where the foreground quantity ratio is less than or equal to the background quantity ratio to obtain a second labeling value; the first labeling value and the second labeling value are different; and determining the image feature mask according to the first labeling value and the second labeling value.

[0012] In one embodiment, the sampling features and the features to be identified are input into a pre-trained attention network for feature extraction processing to obtain target image features, including: performing linear transformation processing on the sampling features to obtain a query vector; performing linear transformation processing on the features to be identified to obtain a key vector and a value vector; performing feature extraction processing on the query vector, the key vector and the value vector through the attention network to obtain the target image features.

[0013] In one embodiment, determining the corresponding target text according to the target image features includes: performing feature extraction processing on the prompt text according to the received prompt text to obtain prompt text features; performing feature splicing processing on the prompt text features and the target image features to obtain spliced ​​features; and inputting the spliced ​​features into a pre-trained large language model to obtain the target text output by the large language model.

[0014] The second aspect of the present application provides a text recognition device, including: a text prediction module, used to perform text segmentation processing on an image to be recognized, and obtain a segmented image containing a text area; a mask generation module, used to perform binary labeling processing on the segmented image according to pixel information of the text area, and obtain an image feature mask; a sampling module, used to perform sampling processing on the features to be recognized of the image to be recognized according to the image feature mask, and obtain sampling features; a feature extraction module, used to input the sampling features and the features to be recognized into a pre-trained attention network for feature extraction processing, and obtain target image features; a text determination module, used to determine the corresponding target text according to the target image features.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned text recognition method.

[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned text recognition method when the program instructions are executed by a processor.

[0017] The above scheme can obtain a segmented image containing the text area after segmentation by performing text segmentation on the image to be recognized; perform binary labeling on the segmented image according to the pixel information of the text area to obtain an image feature mask with foreground and background separation; perform sampling processing on the features to be recognized of the image to be recognized according to the image feature mask, and then sample the sampling features of the text area in the image to be recognized; input the sampling features and the features to be recognized into the pre-trained attention network for feature extraction processing, and obtain the target image features output by the attention network, and then determine the corresponding target text according to the target image features. In this way, the amount of data processing of the neural network in the subsequent process can be reduced through the image sampling processing method, and the accuracy of text recognition can also be improved.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1 is a flowchart of an exemplary embodiment of the text recognition method of the present application; Figure 2 is an exemplary text font de-stylization schematic diagram in the text recognition method of the present application; Figure 3 is a schematic diagram of an exemplary image semantic block in the text recognition method of the present application; Figure 4 is a schematic diagram of an exemplary text recognition model in the text recognition method of the present application; Figure 5 is a block diagram of a text recognition device shown in an exemplary embodiment of the present application; Figure 6 It is a structural schematic diagram of an embodiment of the electronic device of the present application; Figure 7 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0021] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0022] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0023] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] For ease of understanding, one of the applicable scenarios of the present application is now exemplarily described.

[0025] Optical character recognition technology (OCR) has a large number of usage scenarios in real life. However, due to differences in character styles, fonts, and the influence of various external environments, it is easy for the text recognition model to experience performance degradation or misrecognition. In recent years, the Large Language Model (LLM) technology based on the Transformer architecture has made rapid progress. On this basis, the computer vision field has used the same Vision Transformer method and its global attention mechanism to greatly improve the versatility and robustness of various methods in the visual field. Therefore, the large language model based on the Transformer architecture can be used in text recognition methods.

[0026] However, the amount of computation of the self-attention mechanism is proportional to the square of the number of tokens. Too many tokens can easily make model training and model reasoning more difficult and cost more. Among them, token refers to the smallest semantic unit that represents a word in a language model. Before the prompt word text is sent to the neural network, the Tokenizer will decompose long texts such as compound words, sentences, paragraphs, and articles into the smallest unit of token tokens, and then convert the tokens into vector representation data structures through Embedding, and finally input them into the neural network. It is understandable that when recognizing text in an image, it is necessary to decompose the image into image tokens, and then input the feature vectors of the image tokens into the neural network to determine the text information therein.

[0027] In everyday scenes, we can observe that the number of words in different images is randomly distributed: for example, there may be a large number of advertising slogans in shopping mall images, while it is often difficult to see text slogans in remote outdoor scenes. As image acquisition equipment continues to upgrade, its imaging resolution is getting better and better, and the information contained in the image is also becoming more diverse and complex.

[0028] Therefore, using the same text recognition method to recognize text in images of different scenes does not meet the actual needs. How to adaptively recognize the text in an image based on the amount of information already contained in the image is of certain significance for text recognition in natural scenes.

[0029] See also Figure 1 , Figure 1 This is a flowchart of an exemplary embodiment of the text recognition method of the present application. Specifically, it may include the following steps: Step S110, performing text segmentation processing on the image to be recognized to obtain a segmented image containing a text area.

[0030] Performing text prediction processing on the image to be recognized refers to classifying each pixel in the image to be recognized and predicting whether it represents a text area in the image to be recognized based on its category.

[0031] For example, in this application, a segmentation network TextSegNet can be used to perform text segmentation processing on the image to be recognized to obtain a segmented image Specifically, the image to be identified can be input as the input image of the segmentation network, and then the Seg-Encoder in the segmentation network downsamples and extracts features of the image to be identified to obtain the deep features of the image to be identified. ; Then the Seg-Decoder in the segmentation network upsamples the extracted deep features to obtain the segmented image .

[0032] The core principle of SegNet is to achieve semantic segmentation of images through deep learning methods. Unlike traditional pixel-level classification methods, SegNet can not only classify each pixel in the image, but also assign each pixel to a predefined category, thereby achieving refined segmentation of the image, thereby predicting the text area and non-text area in the image to be identified. Among them, the encoder is responsible for extracting deep features from the original image, while the decoder uses these features to generate the final semantic segmentation map. The Seg-Encoder encoder in the segmentation network can use a stacked Convnet to downsample and extract features from the image to be identified. The decoder gradually restores the resolution through upsampling and deconvolution operations, and reconstructs the semantic segmentation of the input image based on the features extracted by the encoder. The details are not described here.

[0033] It should be noted that, when executing the above method, the features to be identified of the image to be identified may also be Downsampling, feature extraction, and upsampling are performed in the input segmentation network.

[0034] Optionally, the text area and non-text area in the image to be identified can be determined by using a pixel-based segmentation method in image processing (such as based on the image's pixel information, edge information, connected domain information, etc.) to obtain a segmented image, which is not limited here.

[0035] Step S120, performing binary labeling processing on the segmented image according to the pixel information of the text area to obtain an image feature mask.

[0036] Combining the above steps, it can be seen that by segmenting the image to be identified, the segmented image can be obtained. , and according to the segmented image The pixel information of each pixel in the image (such as the category of each pixel) can be used to determine the foreground and background in the segmented image, which is equivalent to determining the text area and non-text area in the segmented image.

[0037] The binary labeling process of the present application may be labeling the segmented image using 0 and 1, or labeling the segmented image using two other different values, which is not limited here.

[0038] Specifically, the pixels in the foreground of the segmented image can be marked as 1 and the pixels in the background can be marked as 0; or the segmented image can be partitioned to mark the area with more foreground pixels as 1 and the area with more background pixels as 0, which is not limited here. In this way, the text area and the non-text area can be segmented and represented, realizing the segmentation based on the image. The pixel information in determines the image feature mask .

[0039] Step S130 , sampling the features to be identified of the image to be identified according to the image feature mask to obtain sampled features.

[0040] The above steps are described below. After the above steps are processed, the image feature mask can represent the distinction between the text area and the non-text area of ​​the image to be identified.

[0041] Therefore, in this step, the image feature mask can be As the weight of the sampling process, the features to be identified in the image are treated accordingly according to the image feature mask. Perform sampling processing to obtain sampling characteristics .

[0042] It can be understood that after the image to be identified is sampled using the image feature mask in this step, the obtained sampling features significantly simplify the information contained in the image compared to the image to be identified.

[0043] Step S140, input the sampled features and the features to be identified into a pre-trained attention network for feature extraction processing to obtain the target image features.

[0044] The attention mechanism can help the neural network model better focus on the key information in the input data. For example, in the image recognition scenario, the attention mechanism can help the neural network model better focus on the key features in the image, thereby improving the recognition accuracy and allowing more accurate text to be obtained later.

[0045] Combined with the above steps, the sampled features are used as the query subject and the features to be identified of the image to be identified are used as the query object to be input into the pre-trained attention network for feature extraction processing, then the target image features output by the attention network can be obtained. .

[0046] Step S150, determining the corresponding target text according to the target image features.

[0047] In conjunction with the above steps, after obtaining the target image features, the target image features can be input into the trained large language model to obtain the target text output by the large language model. Alternatively, similarity calculation is performed based on the target image features and the base image features in the preset database, and the target base image features that are most similar to the target image features in the base image features are determined by feature similarity. Each base image feature can have corresponding text information of a character, word or sentence, so that the corresponding target text can be found and output according to the target image features.

[0048] It can be seen that the present application can obtain a segmented image containing a text area after segmentation by performing text segmentation processing on the image to be recognized; perform binary labeling processing on the segmented image according to the pixel information of the text area to obtain an image feature mask with foreground and background separation; perform sampling processing on the features to be recognized of the image to be recognized according to the image feature mask, and then sample the sampling features of the text area in the image to be recognized; input the sampling features and the features to be recognized into a pre-trained attention network for feature extraction processing, and obtain the target image features output by the attention network, and then determine the corresponding target text according to the target image features. In this way, the amount of data processing of the neural network in the subsequent process can be reduced through the image sampling processing method, and the accuracy of text recognition can also be improved.

[0049] Based on the above embodiments, the embodiment of the present application describes the steps of sampling and processing the features to be identified of the image to be identified according to the image feature mask to obtain the sampled features. The image feature mask includes a foreground area and a background area. Specifically, the method of this embodiment includes the following steps: The sampling times are determined according to the number of foreground areas; discrete sampling is performed on the features to be identified of the image to be identified according to the sampling times to obtain sampling features.

[0050] Combined with the above embodiment, after obtaining the image feature mask In the process of , the foreground area and the background area are marked by binary labeling. Among them, the foreground area is equivalent to the text area determined by the segmentation network, that is, it corresponds to the area that needs to be sampled in the image to be recognized.

[0051] Therefore, the sampling times can be determined according to the number of foreground areas in the image feature mask (for example, the number of foreground areas is determined as the sampling times, or a value less than or greater than the number of foreground areas is determined as the sampling times), which is equivalent to adaptively determining the sampling times according to the number of targets contained in the image feature mask. In the case of binary labeling with 0 and 1, the number of targets is equivalent to (Sum the label values ​​of each foreground and background area in the image feature mask).

[0052] For example, the discrete sampling process of the feature to be identified can be performed using a matrix-shaped image feature mask. Features to be identified in the image to be identified Discrete sampling or other mathematically differentiable sampling methods are performed, which are not limited here. Then the sampling characteristics after adaptive sampling can be obtained . The mathematical expression can be:

[0053] in, Represents a multinomial distribution, using As the weight of multinomial distribution sampling, the target of potential preceding text area in the image to be recognized is sampled. R refers to the set of real numbers. Represents the feature dimension of each image token after mapping. m refers to the image feature mask determined by partitioning the segmented image. For example, using the preset Pixel size window to H*W size text area By performing non-overlapping partition processing, we can obtain partitions.

[0054] Based on the above embodiments, the present embodiment of the application describes the steps before performing text segmentation processing on the image to be recognized and obtaining a segmented image containing the text area. Specifically, the method of this embodiment includes the following steps: The received original image is resized to obtain an image to be processed that matches a preset size; the image to be processed is segmented and embedded to obtain an image to be identified and features to be identified of the image to be identified.

[0055] In conjunction with the above embodiments, the image to be recognized in the present application may be an image that has been processed or an image that has not been processed. The process of performing text prediction processing on the image to be recognized may be to input the image to be recognized into a segmentation network for text prediction processing; or to input the features to be recognized of the image to be recognized into a segmentation network for text prediction processing.

[0056] Taking the method of inputting the features to be recognized into the segmentation network for text prediction processing as an example, it is necessary to obtain the features to be recognized before performing text prediction processing on the image to be recognized.

[0057] For example, the image to be identified before image processing is called the original image. The original image is resized to obtain an image to be processed I matching the preset size (H*W), which can be recorded as , for subsequent image processing. The window is used to segment and embed the image I to be processed (Patch Embedding). The principle is to use the above window to segment the image I to obtain the image to be identified, and then embed the image to be identified into the vector space, and record the image tokens obtained after encoding as , then the features to be identified of the image to be identified are obtained. Among them, Indicates the number of tokens. Indicates the feature dimension of each image token after mapping, which can be set to 512 according to experience.

[0058] Therefore, the subsequent process can be to Input segmentation network TextSegNet, through the segmentation network TextSegNet Seg-Encoder (can use stacked Convnet) Further downsampling and feature extraction result in a size of Feature map , and then Seg-Decoder upsamples the feature map to obtain the predicted segmentation image , whose size is In this embodiment, the segmentation network TextSegNet is mainly used to perform binary classification on the image I, and the category of each pixel in the obtained segmentation map can represent whether it is a text area.

[0059] Based on the above embodiments, the embodiment of the present application exemplarily illustrates the steps of performing text segmentation processing on the image to be recognized to obtain a segmented image containing a text area. Specifically, the method of this embodiment includes the following steps: The features to be identified of the image to be identified are input into a pre-trained image segmentation network for segmentation processing and font standardization processing to obtain a segmented image; the font types of each text area in the segmented image are the same.

[0060] The segmentation process can refer to the description of the previous and next embodiments, and will not be described here in detail. The font standardization process refers to the texts of different font types (such as Kaiti, Songti, etc.) that may exist in the image to be recognized.

[0061] It should be noted that, considering the diversity and complexity of text fonts in real scenes, although continuously expanding training data is a conventional and effective method (which can also be selectively implemented in this application), continuous data collection, annotation and cleaning require huge resources. Therefore, during the training process, the segmentation network of this application does not need to directly learn to generate the corresponding various text fonts in the input image, but learns the fonts through the standardized binary image. .like Figure 2 As shown, Figure 2 This is an exemplary text font de-stylization schematic diagram in the text recognition method of the present application. For texts in various fonts in an image, they can all be processed into a font style type through style standardization.

[0062] Thus, when recognizing and detecting texts in different fonts, it is possible to avoid the font style (type) affecting the final recognition and detection results. Therefore, the image segmentation network of the present application can standardize the font types of each text area in the image to be detected so that the font types of each text area are the same. Specifically, when training the image segmentation network, reference data on font styles can be introduced for model training to achieve the effect of de-stylizing text, which will not be elaborated here.

[0063] For example, in the training phase, the parameters in the Patch Embedding processing module can be adjusted through supervised training so that the neural network can further focus on the character feature information after the font style is removed. and Calculate the binary cross entropy loss. Considering the different difficulty levels of classification for each pixel, when the predicted value approaches 0.5, the loss weight should be increased. From the definition of information entropy, for numbers between 0 and 1, the entropy value is the largest when the value is 0.5. Therefore, the information entropy of each pixel is used as the weight of the binary cross entropy loss. Its mathematical expression can be:

[0064] in express The pixel position in The corresponding predicted value ranges from 0 to 1; express The pixel position in The corresponding label value is 0 or 1.

[0065] The main purpose of text de-stylization is to convert texts of various styles into a certain standard style. Therefore, style loss can be added as an auxiliary constraint. Its mathematical expression can be:

[0066] in It represents the commonly used style loss calculation method, which will not be described here.

[0067] TextSegNet can adapt the structure based on the existing image segmentation network, directly load the corresponding trained model weights, and freeze the model parameters of this part during the training process, reducing the computing resources required for training.

[0068] Through the above method, an auxiliary constraint for font de-stylization is proposed, and the text in the binary label map in the segmentation network is standardized. The shallow features of the network are supervised by style loss and weighted segmentation loss, so that the shallow stage of the network can extract the potential representative features of the characters and reduce the misrecognition caused by font style. As an auxiliary module of the text recognition method in this application, the segmentation network does not require the binary segmentation map to be sufficiently accurate, which can also simplify the difficulty of training and further improve the robustness of the text recognition network.

[0069] Based on the above embodiment, the embodiment of the present application describes the steps of inputting the features to be identified of the image to be identified into a pre-trained image segmentation network for segmentation processing and font standardization processing to obtain a segmented image. Specifically, the method of this embodiment includes the following steps: The features to be identified of the image to be identified are input into a pre-trained image segmentation network for downsampling and feature extraction to obtain downsampled features of the features to be identified; the downsampled features are upsampled through the image segmentation network to obtain a segmented image containing the text area.

[0070] For the specific processing process of this embodiment, reference may be made to the description in the aforementioned embodiment and will not be repeated here.

[0071] Based on the above embodiment, the embodiment of the present application describes the steps of performing binary labeling processing on the segmented image according to the pixel information of the text area to obtain the image feature mask. Specifically, the method of this embodiment includes the following steps: The segmented image is partitioned to obtain at least one image region; according to whether each pixel in each image region is in the text region, the binary label corresponding to each image region is determined to obtain an image feature mask.

[0072] Combined with the above embodiment, the segmented image is obtained. After that, you can use The window size is used to perform non-overlapping partition processing on the segmented image to obtain at least one image region. The size of the segmented image is H*W, so the number of image regions obtained by partitioning can be The predicted classification results of each pixel in each area are counted in turn, and the window corresponding to each image area is binary labeled according to the pixel category of each pixel in each image area (whether it is in the text area) to obtain the image feature mask.

[0073] For example, according to a preset window size, the segmented image is subjected to sliding window statistical processing (determining the type of pixel points in each window of the segmented image), and each window area is binary-labeled according to the statistical results to obtain an image feature mask.

[0074] On the basis of the above embodiments, the embodiment of the present application describes the steps of determining the binary label corresponding to each image area according to whether each pixel point in each image area is in the text area, and obtaining the image feature mask. The segmented image includes a text area and a non-text area, the pixel points in the text area are foreground pixels, and the pixel points in the non-text area are background pixels. Specifically, the method of this embodiment includes the following steps: Obtain the foreground number ratio of foreground pixels and the background number ratio of background pixels in each image area; perform a first marking process on the image area where the foreground number ratio is greater than the background number ratio to obtain a first marking value; perform a second marking process on the image area where the foreground number ratio is less than or equal to the background number ratio to obtain a second marking value; the first marking value and the second marking value are different; determine the image feature mask according to the first marking value and the second marking value.

[0075] In conjunction with the above-mentioned embodiments, the present application performs text segmentation on the image to be recognized, and can obtain the type corresponding to each pixel in the segmented image, that is, characterize whether each pixel is a foreground pixel or a background pixel (equivalent to a text area pixel or a non-text area pixel). According to the binary marking method provided in the above-mentioned embodiments, the first marking value can be 1, and the second marking value can be 0.

[0076] Specifically, when binary labeling is performed on each image region, the labeling method can be determined by referring to the type of pixels in each image region. For example, the predicted classification results of each pixel in each region are counted respectively, and the ratio of the number of foreground pixels to the number of background pixels in each region is calculated. For example: 1. Calculate the ratio of the number of foreground pixels in each region to the number of all pixels in the region to obtain the foreground number ratio. If the foreground number ratio of the region is greater than 50% or greater than other preset ratio thresholds, the region can be marked as the first marking value; if not, the region is marked as the second marking value.

[0077] 2. Calculate the ratio of the number of foreground pixels in each region to the number of all pixels in the region to obtain the foreground number ratio. Similarly, calculate the ratio of the number of background pixels in each region to the number of all pixels in the region to obtain the background number ratio. Mark the region where the foreground number ratio is greater than the background number ratio as the first marking value; mark the region where the foreground number ratio is less than or equal to the background number ratio as the second marking value.

[0078] The segmented image is The area with a large proportion of foreground is marked as 1, otherwise it is marked as 0, thus obtaining a binary labeling matrix, which is also the image feature mask. .

[0079] Based on the above embodiments, the present embodiment of the present application describes the steps of inputting the sampled features and the features to be identified into a pre-trained attention network for feature extraction processing to obtain the target image features. Specifically, the method of this embodiment includes the following steps: The sampled features are linearly transformed to obtain a query vector; the features to be identified are linearly transformed to obtain a key vector and a value vector; the query vector, key vector and value vector are feature extracted through an attention network to obtain the target image features.

[0080] In combination with the above-mentioned embodiment, in the process of extracting the sampling features and the features to be identified based on the attention network, the sampling features need to be After linear transformation, we get the query vector Query (Q), and the features to be identified Perform linear transformation to obtain the key vector Key (K) and value vector Value (V). Then process based on the Q, K, V vectors and the cross attention mechanism to obtain the target image features. , its mathematical expression can be:

[0081] in, , and Represents a learnable transformation matrix. Then repeat the Group Query Attention calculation N-1 times (N can be preset based on daily experience, not limited here), and the final output image feature r can be obtained. .

[0082] Based on the above embodiments, the present embodiment of the present application describes the steps of determining the corresponding target text according to the target image features. Specifically, the method of this embodiment includes the following steps: According to the received prompt text, feature extraction is performed on the prompt text to obtain prompt text features; the prompt text features and target image features are spliced ​​to obtain spliced ​​features; the spliced ​​features are input into a pre-trained large language model to obtain the target text output by the large language model.

[0083] Prompt text refers to text information used to prompt the large language model (LLM), which can be used to instruct or guide the large language model to produce specific outputs. In order for the large language model to accurately output the required data, the acquired target image features and prompt text can be input into the large language model together, so that the large language model can obtain information from the target image features, generate information and output it according to the instructions of the received prompt text to obtain the target text.

[0084] For example, the text prompt may be "Please output all texts in the image in sequence according to the semantic block information, and use semicolons as separators for different semantic block texts". The semantic block may be a slogan or slogan in the corresponding image, and different semantic blocks often have more intuitive differences in font, size, color, position, etc. Figure 3 As shown, Figure 3 This is an exemplary image semantic block diagram in the text recognition method of the present application. AB and CD belong to the same semantic block, EFG belongs to the same semantic block, HI and JK belong to the same semantic block, and LMN belongs to the same semantic block. Therefore, after the large language model analyzes and processes the above text prompts and image semantic blocks, the output target text can be "ABCD; EFG; HIJK; LMN".

[0085] Optionally, in addition to using the prompt text and the large language model to generate the target text, other text generation models can be used or the corresponding target text can be directly generated according to the target image features by feature similarity search, which is not limited here.

[0086] Specifically, the text prompt can be processed by the text feature extractor to obtain the prompt text feature, which is recorded as , and the target image features After splicing, the splicing features are obtained and used as the input of LLM, and then the target text output by LLM is obtained.

[0087] It should be noted that you can refer to Figure 4 As shown, Figure 4 is a schematic diagram of an exemplary text recognition model in the text recognition method of the present application. When implementing the text recognition method of the present application, the application process of the model is mainly: scaling the original image to The size of the image is obtained through the Patch Embedding operation of the segmentation embedding module, and then sent to the lightweight segmentation network TextSegNet and the image encoder Vision Transformer Encoder to obtain the image feature mask and the features to be identified. Among them, the image encoder can include a cross attention mechanism module (CrossAttention) and a group query attention module (Group Query Attention). The adaptive sampling module performs sampling processing to determine the filtered encoding features (which can be recorded as dynamic image tokens). Finally, the encoding features of the prompt text (which can be recorded as text tokens) are combined and sent to the large language model LLM to generate the corresponding prediction results.

[0088] In the model training process of this application, when the concatenated features are input into LLM for training, after LLM forward calculation, the loss calculation of the predicted result of the large language model and the data annotation can be performed, which is recorded as Then the mathematical expression of the overall loss of the text recognition network of this application can be:

[0089] in , and is the weight parameter of the loss, which can be set according to the empirical value , , , which is not limited here. Among them, since the segmentation network is an auxiliary module of the text recognition model of the present application, the segmentation map output by the segmentation network is not required to be sufficiently accurate, so less weight can be used for the loss of the segmentation network part. In the training stage, the parameters of the trainable part of the model are continuously updated by minimizing the total loss value until the model converges, and a trained neural network is obtained and used to execute the above text recognition process embodiment.

[0090] On the basis of the above embodiments, this embodiment can also provide a method for perceiving environmental information in an image to be identified (or an initial image). Exemplarily, after obtaining the image feature mask in the above embodiments, the image feature mask can be inverted, for example, the value marked as 1 in the image feature mask is inverted to 0, and the value marked as 0 in the image feature mask is inverted to 1, to obtain a background image mask. According to the description of the above embodiments, the background image mask obtained after inverting the image feature mask can shield the information of the text area in the image. Using the background image mask to shield the image to be identified can also refer to the above sampling method to obtain the background image. The background image is input into a pre-trained environment perception network, and the environment perception network extracts the environmental feature information of the background image, and the corresponding target text is determined based on the environmental feature information and the target image features (or the corresponding target text is determined based on the environmental feature information, the target image features and the prompt text features). For details, please refer to the description of the above embodiments, which will not be repeated here.

[0091] Therefore, in the scenario where environmental feature information is introduced, the large language model can fully interpret the image to be recognized, and the target text output by the large language model can contain richer text information. It should be noted that the large language model can not only output environmental information in text form, such as a text description of the environment of the current image, or convert environmental information into a text description form and output it in combination with the recognized text information. It can also be used to deeply understand the image to be recognized with the help of environmental information. For example, the text to be recognized (ABCD, EFG) is on a billboard, while the text to be recognized (HIJK) is on a posting board, and the text to be recognized (LMN) in the image to be recognized is on a banner. By sampling the text area through the image feature mask, its text information (such as ABCD, EFG, HIJK, LMN) can be accurately obtained, and by sampling the non-text area through the background image mask, its environmental information (such as billboards, posting boards, banners, etc.) can be obtained. Therefore, when the large language model receives the prompt text "Please output the text on the banner", it can accurately output "LMN". For another example, when the large language model receives the prompt text "Please output the text on the banner, separated by semicolons", it can accurately output "ABCD; EFG". In addition, it is also possible to identify whether the text to be recognized in the current scene is indoors and / or outdoors, and more scenes where the text is located, based on environmental feature information, which are not limited here.

[0092] On the basis of the above embodiments, this embodiment can also provide a method for determining a text area, including: obtaining contrast information of an image to be identified; in response to the contrast information being higher than a preset contrast threshold, a preset edge detection algorithm or a connected domain detection algorithm can be selected to determine the text area in the image to be identified; in response to the contrast information being lower than or equal to the contrast threshold, the method of using a segmentation network in the above embodiment can be selected to determine the text area in the image to be identified.

[0093] It should be further explained that the execution subject of the text recognition method may be a text recognition device, for example, the text recognition method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the text recognition method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0094] Figure 5 FIG. 1 is a block diagram of a text recognition device shown in an exemplary embodiment of the present application. Figure 5 As shown, the exemplary text recognition device 500 includes: a text prediction module 510, a mask generation module 520, a sampling module 530, a feature extraction module 540 and a text determination module 550. Specifically: The text prediction module 510 is used to perform text segmentation processing on the image to be recognized to obtain a segmented image containing a text area.

[0095] The mask generation module 520 is used to perform binary labeling processing on the segmented image according to the pixel information of the text area to obtain an image feature mask.

[0096] The sampling module 530 is used to perform sampling processing on the features to be identified of the image to be identified according to the image feature mask to obtain sampling features.

[0097] The feature extraction module 540 is used to input the sampled features and the features to be identified into a pre-trained attention network for feature extraction processing to obtain the target image features.

[0098] The text determination module 550 is used to determine the corresponding target text according to the target image features.

[0099] In this exemplary text recognition device, by performing text segmentation processing on the image to be recognized, a segmented image containing the text area can be obtained; the segmented image is subjected to binary labeling processing according to the pixel information of the text area to obtain an image feature mask for separating the foreground and background; the features to be recognized of the image to be recognized are sampled according to the image feature mask, and the sampling features of the text area in the image to be recognized can be sampled; the sampling features and the features to be recognized are input into the pre-trained attention network for feature extraction processing, and the target image features output by the attention network are obtained, and the corresponding target text can be determined according to the target image features. In this way, the amount of data processing of the neural network in the subsequent process can be reduced through the image sampling processing method, and the accuracy of text recognition can also be improved.

[0100] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0101] The functions of each module can be found in the text recognition method embodiment, which will not be described in detail here.

[0102] See also Figure 6 , Figure 6 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 100 includes a memory 101 and a processor 102, and the processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above-mentioned text recognition method embodiments. In a specific implementation scenario, the electronic device 100 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 100 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0103] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above-mentioned text recognition method embodiments. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.

[0104] In this exemplary electronic device, by performing text segmentation on the image to be recognized, a segmented image containing the text area can be obtained; the segmented image is binary labeled according to the pixel information of the text area to obtain an image feature mask with foreground and background separation; the features to be recognized of the image to be recognized are sampled according to the image feature mask, and the sampling features of the text area in the image to be recognized can be sampled; the sampling features and the features to be recognized are input into the pre-trained attention network for feature extraction, and the target image features output by the attention network are obtained, and the corresponding target text can be determined according to the target image features. In this way, the amount of data processing of the neural network in the subsequent process can be reduced through the image sampling processing method, and the accuracy of text recognition can also be improved.

[0105] See also Figure 7 , Figure 7 The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor, and the program instructions 111 are used to implement the steps in any of the above-mentioned text recognition method embodiments.

[0106] In this exemplary storage medium, by running the program instructions in the storage medium, the image to be recognized is subjected to text segmentation processing, and a segmented image containing a text area can be obtained after segmentation; the segmented image is subjected to binary labeling processing according to the pixel information of the text area, and an image feature mask with foreground and background separation is obtained; the features to be recognized of the image to be recognized are sampled according to the image feature mask, and the sampling features of the text area in the image to be recognized can be sampled; the sampling features and the features to be recognized are input into a pre-trained attention network for feature extraction processing, and the target image features output by the attention network are obtained, and the corresponding target text can be determined according to the target image features. In this way, the amount of data processing of the neural network in the subsequent process can be reduced through the image sampling processing method, and the accuracy of text recognition can also be improved.

[0107] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0108] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0109] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0110] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A text recognition method, characterized in that: The method comprises: Perform text segmentation processing on the image to be recognized to obtain a segmented image containing the text area; Performing binary labeling processing on the segmented image according to pixel information of the text area to obtain an image feature mask; Sampling the features to be identified of the image to be identified according to the image feature mask to obtain sampling features; Inputting the sampled features and the features to be identified into a pre-trained attention network for feature extraction processing to obtain target image features; The corresponding target text is determined according to the target image feature.

2. The method according to claim 1, characterized in that The image feature mask includes a foreground area and a background area, and the sampling process of the to-be-recognized features of the to-be-recognized image according to the image feature mask to obtain the sampled features includes: Determining the number of sampling times according to the number of the foreground areas; The features to be identified of the image to be identified are discretely sampled according to the sampling times to obtain the sampling features.

3. The method according to claim 1, characterized in that Before performing text segmentation processing on the image to be recognized to obtain a segmented image containing a text area, the method further includes: The received original image is resized to obtain an image to be processed that matches a preset size; The image to be processed is segmented and embedded to obtain the image to be identified and the features to be identified of the image to be identified.

4. The method according to claim 1, characterized in that The text segmentation process is performed on the image to be recognized to obtain a segmented image containing the text area, including: The features to be identified of the image to be identified are input into a pre-trained image segmentation network for segmentation processing and font standardization processing to obtain the segmented image; the font types of the various text areas in the segmented image are the same.

5. The method according to claim 4, characterized in that The step of inputting the to-be-recognized features of the to-be-recognized image into a pre-trained image segmentation network for segmentation processing and font standardization processing to obtain the segmented image comprises: Inputting the to-be-recognized features of the to-be-recognized image into a pre-trained image segmentation network for downsampling and feature extraction processing to obtain downsampled features of the to-be-recognized features; The downsampled features are upsampled through the image segmentation network to obtain a segmented image containing the text area.

6. The method according to claim 1, characterized in that The step of performing binary labeling processing on the segmented image according to the pixel information of the text area to obtain an image feature mask comprises: Performing partition processing on the segmented image to obtain at least one image region; According to whether each pixel in each image area is in the text area, a binary label corresponding to each image area is determined to obtain the image feature mask.

7. The method according to claim 6, characterized in that The segmented image includes the text area and the non-text area, the pixels in the text area are foreground pixels, and the pixels in the non-text area are background pixels, and the binary label corresponding to each image area is determined according to whether each pixel in each image area is in the text area to obtain the image feature mask, including: Obtaining a foreground quantity ratio of the foreground pixels and a background quantity ratio of the background pixels in each image region; Performing a first marking process on the image area where the proportion of the foreground quantity is greater than the proportion of the background quantity, to obtain a first marking value; Performing a second marking process on the image area where the foreground quantity ratio is less than or equal to the background quantity ratio to obtain a second marking value; the first marking value and the second marking value are different; The image feature mask is determined based on the first label value and the second label value.

8. The method according to claim 1, characterized in that The step of inputting the sampled features and the features to be identified into a pre-trained attention network for feature extraction to obtain target image features includes: Performing linear transformation on the sampled features to obtain a query vector; Performing linear transformation on the feature to be identified to obtain a key vector and a value vector; The query vector, the key vector and the value vector are subjected to feature extraction processing by the attention network to obtain the target image features.

9. The method according to claim 1, characterized in that: The determining the corresponding target text according to the target image feature includes: According to the received prompt text, feature extraction processing is performed on the prompt text to obtain prompt text features; Performing feature splicing processing on the prompt text feature and the target image feature to obtain a splicing feature; The concatenated features are input into a pre-trained large language model to obtain a target text output by the large language model.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images