Text detection method, electronic device, medium and computer program product

Abstract

The present disclosure provides a text detection method, electronic device, medium and computer program product, which relate to the field of data processing technology. The method includes: obtaining image features of a target object; inputting the image features into a first decoder of a text detection model to obtain a first feature, and obtaining a first text detection result based on the first feature; inputting the first feature into a second decoder of a text detection model to obtain a second feature, and obtaining a second text detection result based on the second feature; wherein the first text detection result and the second text detection result are text detection results of different levels, and the levels of the text detection results include: word level, line level, or paragraph level. The present invention can obtain multiple levels of text detection results at the same time, and does not need to perform text detection on another level based on the text detection result of one level, but obtains the text detection results of each level based on the first feature, and the text detection results are highly accurate.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a text detection method, electronic equipment, medium and computer program product. Background Art

[0002] Text detection is very important in the fields of computer vision and natural language processing. Specifically, through text detection, text information in objects such as images and digital documents can be obtained to perform computer vision processing and natural language processing based on the text information.

[0003] In the related art, images, digital documents, etc. are detected by using word-level, line-level, or paragraph-level text detectors to obtain word-level, line-level, or paragraph-level text detection results.

[0004] However, according to the method of the related technology, only a single level of text detection results can be obtained, and in the related technology, it is necessary to obtain the text detection results of the current level based on the text detection results of the previous level, which will cause the text detection results of the current level to have a low detection result accuracy problem. Summary of the invention

[0005] The embodiments of the present disclosure provide a text detection method, an electronic device, a medium and a computer program product to solve the problem in the prior art that multi-level text detection results cannot be obtained and the text detection results have low accuracy.

[0006] In a first aspect, an embodiment of the present disclosure provides a text detection method, including:

[0007] Obtain image features of the target object;

[0008] Inputting the image feature into a first decoder of a text detection model to obtain a first feature, and obtaining a first text detection result based on the first feature;

[0009] The first feature is input into the second decoder of the text detection model to obtain a second feature, and a second text detection result is obtained based on the second feature; wherein the first text detection result and the second text detection result are text detection results of different levels, and the levels of the text detection results include: word level, line level, or paragraph level.

[0010] In a second aspect, an embodiment of the present disclosure provides a text detection device, including:

[0011] A first acquisition module, used to acquire image features of a target object;

[0012] A second acquisition module, configured to input the image feature into a first decoder of a text detection model to obtain a first feature, and obtain a first text detection result based on the first feature;

[0013] The third acquisition module is used to input the first feature into the second decoder of the text detection model to obtain a second feature, and obtain a second text detection result based on the second feature; wherein the first text detection result and the second text detection result are text detection results of different levels, and the levels of the text detection results include: word level, line level, or paragraph level.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0015] Memory, used to store computer programs;

[0016] A processor is used to execute the computer program to implement the method as described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing a computer program; wherein when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, comprising a computer program / instructions, which implement the steps of the method described in the first aspect when executed by a processor.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] In the disclosed embodiment, the text detection model according to the embodiment can obtain at least two levels of text detection results. Compared with the single-level text detection method of the related art, the text detection result obtained by the embodiment can provide multi-level text information, and the semantic information obtained is more comprehensive, which solves the problem that the multi-level text detection results cannot be obtained in the related art. In addition, in the embodiment, the first feature obtained by the first decoder is used to obtain the first text detection result, and the first feature output by the first decoder is used as the input of the second decoder to obtain the second feature, and then the second text detection result is obtained according to the second feature. The first text detection result and the second text detection result are text detection results of different levels. In the embodiment, the second text detection result does not depend on the first text detection result, which avoids the problem that the first text detection result has errors, resulting in low accuracy of the second text detection result. The semantic information contained in the first feature is richer than the voice information contained in the first text detection result. Therefore, compared with the method of obtaining the second text detection result through the first text detection result, the second text detection result is obtained through the first feature, and the second text detection result is more accurate. That is, this embodiment can obtain text detection results of multiple levels at the same time, and the text detection results of one level do not depend on the text detection results of another level. The text detection results are highly accurate, which solves the problem in the related art that the prediction results of the next level are predicted based on the prediction results of the previous level, resulting in low accuracy of the prediction results.

[0021] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for describing the embodiments are briefly introduced below.

[0023] Figure 1 A step diagram of a text detection method provided by an embodiment of the present disclosure;

[0024] Figure 2 A flowchart of an image feature extraction method provided by an embodiment of the present disclosure;

[0025] Figure 3 A flowchart of another image feature extraction method provided by an embodiment of the present disclosure;

[0026] Figure 4 A schematic diagram of the structure of a predictor provided by an embodiment of the present disclosure;

[0027] Figure 5 A schematic diagram of a text detection result provided by an embodiment of the present disclosure;

[0028] Figure 6 A structural diagram of a text detection model provided in an embodiment of the present disclosure;

[0029] Figure 7 A schematic diagram of the structure of a row detection module provided in an embodiment of the present disclosure;

[0030] Figure 8 A structural diagram of a word detection module provided by an embodiment of the present disclosure;

[0031] Fig. 9 A schematic diagram of the structure of a paragraph detection module provided in an embodiment of the present disclosure;

[0032] Fig.10 A schematic diagram of a text detection device provided in an embodiment of the present disclosure;

[0033] Fig.11 A block diagram of an electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can also be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0035] The following explanations apply to some nouns or terms appearing in this application:

[0036] Region of Interest (ROI) refers to the area that needs special attention that is marked out from the processed image in the form of a box, circle, ellipse, irregular polygon, etc. in the field of machine vision and image processing.

[0037] Visual Question Answering (VQA) is a technology that extracts text descriptions of questions from images and predicts answers based on the questions.

[0038] It should be noted that the sample data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0039] Figure 1 is a step diagram of a text detection method provided by an embodiment of the present disclosure, including:

[0040] Step 101: Acquire image features of a target object.

[0041] The target object can be an image, a digital document, or other objects that require text detection. Image features are extracted from the target object to obtain the image features of the target object.

[0042] For example, the target object for text detection may be input into a feature encoder in a text detection model to obtain image features of the target object.

[0043] The target object may be preprocessed, and then the image features of the preprocessed target object may be extracted. Further, the preprocessed target object may be input into a feature encoder in a text detection model to obtain image features generated by the feature encoder.

[0044] For example, the method for preprocessing the target object may include at least one of the following: randomly flipping the target object, randomly rotating the target object within a preset angle range (such as -10° to 10°), randomly scaling the target object within a preset zoom factor range (such as 0.5 times to 3 times), and randomly cropping the target object.

[0045] After preprocessing the target object, the target object is adjusted to a preset size, and the resized target object is input into the text detection model, and the image features of the target object are obtained through the feature encoder in the text detection model. The preset size can be a size that is compatible with the text detection model, for example, the preset size can be 640×640.

[0046] For example, the feature encoder may include a convolutional neural network and a feature encoder, and the image features of the target object are extracted through the convolutional neural network and the feature encoder.

[0047] For example, refer to Figure 2, a feature encoder composed of a backbone network in a deep residual network (e.g., Deep Residual Network-50, ResNet-50) and an encoder (e.g., Transform encoder) can be used to extract and enhance features of the target object to obtain image features of the target object. The image features may include image features of at least one scale.

[0048] Furthermore, the feature encoder includes a ResNet-50 backbone network and a Transform encoder, and the image features of the target object can be extracted through the ResNet-50 backbone network to obtain feature maps of four scales: C1, C2, C3, and C4. The resolution of the feature map of scale C1 is 1 / 4 of the resolution of the target object; the resolution of the feature map of scale C2 is 1 / 8 of the resolution of the target object; the resolution of the feature map of scale C3 is 1 / 16 of the resolution of the target object; and the resolution of the feature map of scale C4 is 1 / 32 of the resolution of the target object.

[0049] The feature maps of scales C2, C3, and C4 can be mapped into feature maps with 256 channels through a 1×1 convolution layer, and then the feature maps are input into the Transform encoder to obtain image features of different scales.

[0050] The Transform encoder in the related art usually requires a large amount of image data for training, and the convergence speed is slow during the training process. In actual applications, the number of images in the text inspection training set is usually limited, and the synthesized images lack background changes. Therefore, the existence of related technologies makes it difficult to obtain training sets, resulting in high difficulty in training the Transform encoder, and the attention mechanism in the Transform encoder is slow and difficult to converge, and the resolution of extracting image features is limited.

[0051] Reference Figure 3 In order to avoid the problem that the number of text detection training sets cannot meet the training requirements of the Transform visual model in the related technology, the Transform encoder of this embodiment adopts a deformable attention mechanism and performs feature extraction through the deformable attention mechanism and the feedforward network.

[0052] Furthermore, the deformable attention mechanism combines the sparse sampling capability of the Deep Cross Network (DCN) model and the global relationship modeling capability of the Transform encoder. Based on the deformable attention mechanism, each feature pixel of the target object does not need to interact with all other feature pixels, but only needs to interact with some pixels obtained based on sampling, and the positions of these sampling points are learnable. The deformable attention mechanism is an efficient attention mechanism for local and relational modeling and sparse sampling. Based on this mechanism, the problems of slow convergence speed of the attention mechanism of the Transform encoder and limited feature resolution in related technologies can be solved.

[0053] Furthermore, the Transform encoder includes multiple encoding units, each of which includes a Multi-Scale Deformable Attention (MSDA) layer and a FeedForward Network (FFN). The Transform encoder aggregates the contextual information in the image features to output enhanced multi-scale feature maps C2+, C3+, and C4+, and upsamples the enhanced multi-scale image feature maps C2+, C3+, and C4+ to obtain a high-resolution feature map C1+.

[0054] Specifically, C1+, C2+, C3+, and C4+ represent feature maps of different scales, and the image features of C1+, C2+, C3+, and C4+ are expressed as , , , .

[0055] in, and Represents the height and width of the image respectively. Indicates the number of channels, Indicates characteristics. , , and Represent image features of different scales, , , and Represent image features of different scales. For example, the number of channels .

[0056] Step 102: input the image feature into a first decoder of a text detection model to obtain a first feature, and obtain a first text detection result based on the first feature.

[0057] Specifically, the image feature is input into the first decoder of the text detection model, the first feature is obtained according to the image feature and the attention operation mechanism of the first decoder, the first feature is input into the first predictor corresponding to the first decoder, and the first text detection result is obtained.

[0058] Furthermore, the image features are input into the first decoder of the text detection model, and based on the attention operation mechanism of the first decoder, a query feature is obtained, and the query feature is determined as the first feature in this step.

[0059] Step 103: input the first feature into a second decoder of the text detection model to obtain a second feature, and obtain a second text detection result based on the second feature.

[0060] Specifically, the first feature is input into the second decoder of the text detection model, and a feature is obtained according to the attention operation mechanism of the first feature and the second decoder, and the query feature is determined as the second feature of this step. The second feature is input into the second predictor corresponding to the second decoder to obtain a second text detection result.

[0061] The first text detection result and the second text detection result are text detection results at different levels, and the levels of the text detection results include: word level, line level, or paragraph level.

[0062] For example, the first text detection result includes text detection results of one level, and the second text detection result includes text detection results of one or two levels other than the level of the first text detection result.

[0063] For example, the first text detection result is a line-level text detection result, and the second text detection result may be a word-level text detection result, a paragraph-level text detection result, or both a word-level text detection result and a paragraph-level text detection result.

[0064] For another example, the first text detection result is a word-level text detection result, and the second text detection result is a paragraph-level text detection result.

[0065] In an example, the first decoder is a row decoder, the first feature is a row query feature, and the first text detection result is a row-level text detection result; the second decoder includes a word decoder, the second feature includes the first word query feature, and the second text detection result includes a word-level text detection result; and / or, the second decoder includes a paragraph decoder, the second feature includes the first paragraph query feature, and the second text detection result includes a paragraph-level text detection result.

[0066] For example, the first decoder is a word decoder, the first feature is a first word query feature, and the first text detection result is a word-level text detection result; the second decoder is a paragraph decoder, the second feature is a first paragraph query feature, and the second text detection result is a paragraph-level text detection result.

[0067] Further, the first decoder includes a plurality of cascaded line decoders, and the second decoder includes a plurality of cascaded word decoders, and / or a plurality of cascaded paragraph decoders.

[0068] The image features are input into multiple cascaded row decoders to obtain the first row query features, row prediction data is obtained based on the first row query features, and row-level text detection results are obtained based on the row prediction data; the first row query features are input into multiple cascaded word decoders to obtain the first word query features, row prediction data is obtained based on the first word query features, and row-level text detection results are obtained based on the word prediction data; the first row query features are input into multiple cascaded paragraph decoders to obtain the first paragraph query features, paragraph prediction data is obtained based on the first paragraph query features, and paragraph-level text detection results are obtained based on the paragraph prediction data.

[0069] For example, the text detection model includes a row decoder and a row predictor, the image feature is input into the row decoder, the row query feature is obtained through the attention operation mechanism of the row decoder, and the row query feature is determined as the first feature. The first feature is input into the row predictor to obtain the row-level text detection result.

[0070] For example, the text detection model includes a word decoder and a word predictor, the first feature is input into the word decoder to obtain a first word query feature, and the first word query feature is input into the word predictor to obtain a word-level text detection result.

[0071] For example, the text detection model includes a paragraph decoder and a paragraph predictor, the first feature is input into the paragraph decoder to obtain a first paragraph query feature, and the first paragraph query feature is input into the paragraph predictor to obtain a paragraph-level text detection result.

[0072] Further, see Figure 4 , the line predictor, word predictor and paragraph predictor all include a multilayer perceptron (MLP). The corresponding query features are input into the MLP to obtain the text detection result. Among them, the text detection result includes a text candidate box for reflecting the text area.

[0073] Among them, refer to Figure 4, when the predictor is a row predictor, the query feature is the row query feature, the mask feature is the row mask feature, the text classification score is the text classification score of the row query feature, and the mask feature is the row mask feature. When the predictor is a word predictor, the query feature is the word query feature, the mask feature is the word mask feature, the text classification score is the text classification score of the word query feature, and the mask feature is the word mask feature. When the predictor is a paragraph predictor, the query feature is the paragraph query feature, the mask feature is the paragraph mask feature, the text classification score is the text classification score of the paragraph query feature, and the mask feature is the paragraph mask feature. For example, refer to Figure 5 , input the target object into the text detection model, and obtain three levels of text detection results. The three levels of text detection results are: Figure 5 The word-level text detection results shown on the right side of the text detection model. Figure 5 The line-level text detection results shown below the text detection model, and Figure 5 Paragraph-level text detection results shown above the text detection model.

[0074] In computer vision processing and natural language processing, scene text detection plays a very important role as a bridge for transferring text semantic information between the computer vision processing field and the natural language processing field. The method of this embodiment can be applied to the fields of computer vision processing, natural language processing, etc.

[0075] Furthermore, detecting and understanding text in natural scene images and digital documents is widely used in practical applications, including visual recognition, scene understanding, and text-based visual question answering (VQA). Text can be divided into line, word, and paragraph levels. Compared with words in a word, lines (e.g., phrases and sentences) can express more semantic information, and paragraphs can convey richer contextual semantic information.

[0076] However, in the related art, text detection results are obtained through a single-level text detection method. For example, the encoded features of the input image are decoded into independent text examples, and the text detection results of words, lines or paragraphs are obtained through a single basic scene text detector. These detectors only predict text results at one level and do not explore the contextual relationship between texts at different levels. Therefore, these detectors cannot convey more information that is helpful for scene text understanding. This method has very large limitations in conveying semantic information and cannot reflect the relationship between text instances at different levels. However, in order to more comprehensively convey the semantic information of the text, it is necessary to simultaneously detect text information at multiple levels (including words, lines and paragraphs), and the methods of the related art cannot solve this problem.

[0077] In addition, in the related art, the bounding box of the word / line can be predicted by the Transform structure, and then the sequential relationship between the text boxes can be directly learned based on the visual features and position information after the region of interest (ROI) operation. Specifically, the Transformer-based structure is first used to generate the word / line mask map, and then the association relationship between the word and line mask map is modeled. However, these methods all require supervision through a relationship matrix or index labels, and directly perform relationship modeling based on the word-level / line-level detection results. This processing method needs to rely on the detection results of the previous level to model the contextual relationship, that is, the prediction results of the previous level are used as the input of the model of the next level to obtain the prediction results of the next level. Therefore, the accuracy of the prediction results of the next level is highly limited by the text detection results of the previous level. And this method is difficult to expand to text detection tasks of more than two levels. In addition, the method based on the related art requires a high computational cost when detecting text-intensive images including a large amount of text content.

[0078] In summary, the text detection model according to this embodiment can obtain text detection results of at least two levels. Compared with the single-level text detection method of the related technology, the text detection results obtained by this embodiment can provide multi-level text information, and the obtained semantic information is more comprehensive, which solves the problem that multi-level text detection results cannot be obtained in the related technology.

[0079] In addition, relative to the method of inputting the text detection result of the previous level into the text detector of the current level to obtain the text detection result of the current level, this embodiment obtains the first text detection result through the first feature, obtains the second feature through the first feature and the second decoder, obtains the second text detection result according to the second feature, and the first text detection result and the second text detection result are text detection results of different levels. That is, in this embodiment, the second text detection result does not depend on the first text detection result, which avoids the problem of low accuracy of the second text detection result caused by errors in the first text detection result. In addition, the semantic information contained in the first feature is richer than the phonetic information contained in the first text detection result. Therefore, relative to the method of obtaining the second text detection result through the first text detection result, the second text detection result obtained through the first feature is more accurate.

[0080] That is, this embodiment can obtain text detection results of multiple levels at the same time, and the text detection results of one level do not depend on the text detection results of another level. The text detection results are highly accurate, which solves the problem in the related art that the prediction results of the next level are predicted based on the prediction results of the previous level, resulting in low accuracy of the prediction results.

[0081] In one embodiment, the text detection result includes a text candidate box. In step 102, obtaining the first text detection result according to the first feature may include the following sub-steps:

[0082] Sub-step 1021, inputting the first feature and the image feature into a first predictor to obtain a first target mask feature.

[0083] The first feature and the image feature are input into a first predictor to obtain a first mask map including a plurality of first mask features, wherein the first mask map has a plurality of mask features, and one mask feature corresponds to one pixel in the first mask map. A first target mask feature is obtained from the plurality of first mask features.

[0084] For example, step 1021 may include the following sub-steps:

[0085] Sub-step A1: input the first feature and the image feature into a first predictor to obtain a first text classification score of the first feature and a first mask feature corresponding to the first feature.

[0086] The first text classification score reflects the probability that the first text feature belongs to a text feature, and is positively correlated with the probability that the first feature belongs to a text feature.

[0087] Furthermore, the first feature and the image feature are input into a first predictor to obtain a first text classification score of the first feature and a first mask image corresponding to the first feature, and the feature of the first mask image is determined as a first mask feature corresponding to the first query feature.

[0088] For example, sub-step A1 may include the following sub-steps:

[0089] Sub-step A11, inputting the first feature into the multilayer perceptron in the first predictor to obtain the first query feature.

[0090] For example, the first predictor is Figure 4 As shown, the first feature and image feature are input as Figure 4 The multi-layer perceptron (MLP) shown in the figure obtains the first query feature. Preferably, the image feature is a low-level image feature among multiple levels of image features, wherein different levels correspond to different scales, and the low-level image feature has the largest scale among the image features of multiple levels.

[0091] Sub-step A12, obtaining a first mask feature according to the first query feature.

[0092] Specifically, the first query feature is normalized to obtain a first text classification score of the first feature corresponding to the first query feature.

[0093] For example, after the first query feature is obtained through MLP mapping, the first query feature can be normalized to a first text classification score in the interval [0,1] through a Softmax function normalization function, and the first text classification score is directly output. The text classification score is used to represent the binary classification score of whether the first feature corresponding to the third query feature belongs to the text or not.

[0094] Furthermore, a multiplication operation is performed on the first query feature and the image feature to obtain a first mask feature.

[0095] For example, the first query feature and the bottom-level image feature (such as the image feature with a scale of F1) are subjected to Einstein multiplication to obtain a first mask image of the same size as the first feature, and the first mask image is scaled to the original image size and output. The first mask feature in the first mask image is the first mask feature.

[0096] Sub-step A2: obtaining, from the first mask features, a first target mask feature whose first text classification score is greater than or equal to a first preset score threshold.

[0097] The first preset score threshold may be set according to user requirements. For example, the first preset score threshold may be 0.3.

[0098] Furthermore, the first mask feature has a corresponding first mask prediction score, and the first mask prediction score is positively correlated with the probability that the first mask feature is a text mask feature.

[0099] From the first mask features, a first preliminary mask feature whose first text classification score is greater than or equal to a first preset score threshold is obtained, and then from the first preliminary mask features, a first target mask feature whose first mask prediction score is greater than or equal to a third preset score threshold is obtained. The third preset score threshold can be set according to user needs, for example, it can be set to 0.5.

[0100] Sub-step 1022, obtaining a first text candidate box according to the first target mask feature.

[0101] For example, the first mask feature is a mask feature in the first mask map, and the first target mask feature is selected from the first mask feature, so the first target mask feature is also a mask feature in the first mask map. From the first mask map, the area corresponding to each first target mask feature is obtained, and the area corresponding to the first target mask feature and adjacent to each other is connected to obtain the first text candidate box.

[0102] Furthermore, from the first mask map, the area corresponding to each first target mask feature is obtained, and the area corresponding to the first target mask feature and adjacent in position is connected to obtain the first preliminary text candidate box. The average mask prediction score of all pixels in the first preliminary text candidate box is calculated, and the first preliminary text candidate boxes below the first preset average score threshold are filtered out, and the first preliminary text candidate boxes remaining after filtering are determined as the first text candidate boxes that are finally output. For example, the preset average score threshold can be set according to user needs, for example, it can be set to 0.9.

[0103] In step 103, obtaining a second text detection result according to the second feature may include the following sub-steps:

[0104] Sub-step 1031, inputting the second feature and the image feature into a second predictor to obtain a second target mask feature.

[0105] The second feature and the image feature are input into a second predictor to obtain a second mask map including a plurality of second mask features, and a second target mask feature is obtained from the plurality of second mask features.

[0106] Step 1031 may include the following sub-steps:

[0107] Sub-step B1, inputting the second feature and the image feature into a second predictor to obtain a second text classification score of the second feature and a second mask feature corresponding to the second feature.

[0108] The second text classification score is positively correlated with the probability that the second feature is a text feature.

[0109] Further, the second feature and the image feature are input into a second predictor to obtain a second text classification score of the second feature and a second mask map corresponding to the second feature, and the feature of the second mask map is determined as a second mask feature corresponding to the second feature.

[0110] For example, sub-step B1 may include the following sub-steps:

[0111] Sub-step B11, inputting the second feature into the multilayer perceptron in the second predictor to obtain the second query feature.

[0112] For example, the second predictor is Figure 4 As shown, the second feature is input as Figure 4 The multilayer perceptron shown obtains the second query feature.

[0113] Sub-step B12, obtaining a second mask feature according to the second query feature.

[0114] The second query feature is normalized to obtain a second text classification score of the second feature.

[0115] For example, after the second query feature is obtained through MLP mapping, the second query feature can be normalized to a second text classification score in the interval [0,1] through a Softmax function normalization function, and the second text classification score is directly output. The second text classification score is used to characterize the binary classification score of whether the second feature belongs to the text or not.

[0116] Furthermore, a multiplication operation is performed on the second query feature and the image feature to obtain a second mask feature.

[0117] For example, the second query feature and the bottom-level image feature (such as the image feature with a scale of F1) are subjected to Einstein multiplication to obtain a second mask image of the same size as the second feature, and the second mask image is scaled to the original image size and output. The first mask feature in the first mask image is the first mask feature.

[0118] Sub-step B2: obtaining, from the second mask features, a second target mask feature whose second text classification score is greater than or equal to a second preset score threshold.

[0119] The second preset score threshold, for example, the second preset score threshold may be 0.3.

[0120] Furthermore, the second mask feature has a corresponding second mask prediction score, and the second mask prediction score is positively correlated with the probability that the first mask feature is a text mask feature.

[0121] From the second mask features, a second preliminary mask feature whose second text classification score is greater than or equal to a second preset score threshold is obtained, and then from the second preliminary mask features, a first target mask feature whose second mask prediction score is greater than or equal to a fourth preset score threshold is obtained. The fourth preset score threshold can be set according to user needs, for example, it can be set to 0.5.

[0122] Step 1032: Obtain a second text candidate box according to the second target mask feature.

[0123] For example, the second mask feature is a mask feature in the second mask map, and the second target mask feature is selected from the second mask feature, so the second target mask feature is also a mask feature in the first mask map. From the second mask map, the area corresponding to the second target mask feature is obtained, and the area corresponding to the second target mask feature and adjacent to each other is connected to obtain the second text candidate box.

[0124] Further, from the second mask map, the area corresponding to each second target mask feature is obtained, and the area corresponding to the second target mask feature and adjacent to each other is connected to obtain the second preliminary text candidate frame, and the second average mask prediction score of all pixels in the second preliminary text candidate frame is calculated, and the second preliminary text candidate frames below the second preset average score threshold are filtered out, and the second preliminary text candidate frames remaining after filtering are determined as the second text candidate frames that are finally output. For example, the preset average score threshold can be set according to user needs, for example, it can be set to 0.9.

[0125] In one embodiment, the first decoder is a row decoder. Figure 7 , the first decoder includes at least one line decoder, and the first text detection method may include the following steps:

[0126] Step 201 : obtaining pre-generated row query features and pre-generated row mask features through the current row decoder.

[0127] The step 201 of obtaining the pre-generated row query features may include the following sub-steps:

[0128] Sub-step 2011, if the current row decoder is the first row decoder, the initial row query feature is determined as the pre-generated row query feature.

[0129] The initial row query feature is a row query feature obtained by optimizing the preset row query feature when training the text detection model.

[0130] The preset row query feature may be a row query feature randomly generated according to a fixed size. For example, an N×256 feature is randomly initialized and determined as the preset row query feature, where N is a preset number, which may be 100, 200 or other values ​​and is not limited here.

[0131] When training the text detection model, the preset row query features are continuously optimized according to the prediction results of each training round. After obtaining the trained text detection model, the preset row query features in the text detection model are the initial row query features in this step.

[0132] Obtain the preset row query features. During the model training process, optimize the preset row query features to obtain the initial row query features, which are used as the input data of the first row decoder to ensure the availability of the second row query features output by the first row decoder.

[0133] For example, a target object sample is obtained, the target object sample has annotated line labels, word labels and paragraph labels, and the target object sample and the labels are input into a text detection model to obtain a trained text detection model.

[0134] For example, the text detection model of this embodiment may include at least two of a word decoder, a line decoder, and a paragraph decoder, and each decoder obtains a corresponding text detection result according to the method of the aforementioned embodiment. The missing text detection model includes few text detection modules, and the hierarchical missing text detection model can be trained using fewer annotated data samples through semi-supervised detection to solve the problem of multi-level data annotation difficulties.

[0135] Sub-step 2012: If the current row decoder is the Mth row decoder, the row query feature output by the M-1th row decoder is determined as the pre-generated row query feature.

[0136] Among them, M is greater than 1.

[0137] For example, according to the image features, the self-attention layer, the mask attention layer, and the feedforward network of the M-1th row decoder, the row query features output by the M-1th row decoder are obtained.

[0138] For example, refer to Figure 6 There can be n row decoders in the text detection model, preferably, n=3. The row decoder can be a decoder of Transform structure. For each row decoder, the row decoder collects multi-scale image features, and obtains the row query features corresponding to the row decoder according to the image features corresponding to the row decoder. ,in, Represents row query characteristics, Indicates the number of row queries, which can reflect the number of rows. Indicates characteristics, Indicates the number of channels.

[0139] Wherein, obtaining the pre-generated row mask features in step 201 may include the following sub-steps:

[0140] Sub-step 2013, if the current row decoder is the first row decoder, convolution processing is performed on the image features to obtain pre-generated row mask features.

[0141] For example, the image features include image features of multiple scales, and the image features of the largest scale can be input into the convolutional neural network to obtain image mask features.

[0142] For example, the image features include four scales from F1 to F4, and the image scale decreases from F1 to F4. For example, the image features with a scale of F1 can be input into the convolutional neural network to obtain the image mask features.

[0143] Sub-step 2014, if the current row decoder is the Mth row decoder, the row mask feature obtained according to the row query feature output by the M-1th row decoder is determined as the pre-generated row mask feature.

[0144] Among them, M is greater than 1.

[0145] For example, each row decoder has a corresponding row predictor, and the row query feature output by the M-1th row decoder is input into the row predictor corresponding to the row decoder to obtain a row mask map output by the row predictor. The feature of the row mask map is the row mask feature obtained in this step based on the row query feature output by the M-1th row decoder.

[0146] For example, the M-1th row decoder will obtain the row query feature through the self-attention layer, the mask attention layer, and the feedforward network, and input the row query feature into the row predictor corresponding to the M-1th row decoder to obtain the row mask map output by the row predictor. The mask feature in the row mask map is the row mask feature output by the M-1th row decoder.

[0147] Further, see Figure 7 , the decoder outputs the row query feature, and the row query feature is input into the predictor to obtain the row prediction data. The row prediction data includes the row mask map, and the feature of the row mask map is the row mask feature.

[0148] Step 202, obtain the first row of attention features based on the self-attention mechanism and the pre-generated row query features.

[0149] For example, a self-attention operation is performed on the second row of query features to obtain the first row of attention features.

[0150] Step 203, obtaining the second row query features according to the mask attention mechanism, the first row attention features, the image features, and the pre-generated row mask features.

[0151] Perform masked attention operation on the first row of attention features, image features, and pre-generated row mask features to obtain the second row of query features.

[0152] In this example, the row decoder is a Transform structure decoder, which consists of a self-attention layer, a mask attention layer, and a feedforward network. When processing data, the row decoder adds position embedding to the query and key value of each self-attention and mask attention.

[0153] For example, there are multiple row decoders, and the row query features are gradually refined through multiple row decoders. Further, the row query features of the M-1th row decoder are input into the row predictor corresponding to the M-1th row decoder to obtain the M-1th mask map, the mask features of the M-1th mask map are determined as the row mask features of the M-1th row decoder, the row mask features obtained based on the row query features of the M-1th row decoder are determined as the row mask features pre-generated by the Mth row decoder, and the second row query features of the Mth decoder are obtained based on the row mask features and the mask attention processing mechanism.

[0154] For example, step 203 may include the following sub-steps:

[0155] Sub-step 2031, performs feature fusion on the first row attention feature and the pre-generated row query feature to obtain a first fused feature, and performs mask attention calculation on the first fused feature, the image feature, and the pre-generated row mask feature to obtain a second row attention feature.

[0156] Specifically, the first row attention feature and the pre-generated row query feature are added, and then the added result is normalized to obtain the first fusion feature.

[0157] Sub-step 2032, performing feature fusion on the first fusion feature and the second row of attention features to obtain a second fusion feature, and inputting the second fusion feature into the first feedforward network to obtain a third row of attention features.

[0158] Specifically, the first fusion feature and the second row attention feature are added, and then the result of the addition is normalized to obtain the second fusion feature.

[0159] Sub-step 2033, fusing the third row of attention features and the second fusion features to obtain the second row of query features.

[0160] Specifically, the attention feature of the third row is added to the second fusion feature, and then the result of the addition is normalized to obtain the second fusion feature.

[0161] For example, the image feature includes multiple image features of different scales, the scale of the image feature corresponding to the Mth row decoder, the row decoder has the image feature of the corresponding scale, and is greater than the scale of the image feature corresponding to the M-1th row decoder. Correspondingly, step 203 may include the following steps:

[0162] Sub-step 2034, obtains the second row of query features according to the mask attention mechanism, the first row of attention features, the first row of mask features, and the image features of the scale corresponding to the current row decoder.

[0163] For example, n=3, the image features include four image features from F1 to F4, and the scales of the image features are: scale of F1> scale of F2> scale of F3> scale of F4. Since the aspect ratio of the row is relatively large, the three image features F2 to F4 are used to obtain the row prediction results. Specifically, the image features corresponding to the first row decoder are image features with a scale of F4, the image features corresponding to the second row decoder are image features with a scale of F3, and the image features corresponding to the third row decoder are image features with a scale of F2.

[0164] Step 204: If the current row decoder is the Nth row decoder, the second row query feature is determined as the first feature.

[0165] For example, if the current row decoder is not the Nth row decoder, return to step 201, and re-execute the steps shown in steps 201 to 203 until the second row query feature of the Nth row decoder is obtained, and the second row query feature of the Nth row decoder is determined as the first feature.

[0166] By way of example, the Nth row decoder may be any one of a plurality of row decoders. Preferably, the Nth row decoder is the last one of the plurality of row decoders.

[0167] Preferably, when there are multiple row decoders, the second row query feature output by the last row decoder may be determined as the first feature. When there is only one row decoder, the second row query feature output by the row decoder may be determined as the first feature.

[0168] The second row query feature of the Nth row decoder is determined as the first feature. If the multi-level text detection result includes a word-level text detection result, the word-level text detection result can be accurately obtained according to the second row query feature of the Nth row decoder. If the multi-level text detection result includes a paragraph-level text detection result, the paragraph-level text detection result can be accurately obtained according to the second row query feature of the Nth row decoder. Compared with the method of first obtaining row prediction data and then obtaining word-level text detection results and paragraph text detection results according to the row prediction data, this embodiment can avoid the problem of inaccurate row prediction data leading to inaccurate word-level text detection results and paragraph text detection results. This embodiment improves the accuracy of the text detection results, and obtains word-level text detection results and paragraph-level text detection results according to the first feature. The word-level text detection result can reflect the layout relationship between words and rows, and the paragraph-level text detection result can reflect the layout relationship between paragraphs and rows.

[0169] For example, after step 204, the method further includes:

[0170] Step 205: Input the first feature into the row predictor in the text detection model to obtain the row mask feature of the current row decoder.

[0171] For example, the structure of the row predictor is as follows Figure 4 As shown, the first feature and the low-level image feature are input into the predictor to obtain row prediction data. The low-level image feature is the image feature with the largest scale among the multi-scale image features, and the row prediction data includes the row text classification score and the row mask feature.

[0172] Step 206: If the current row decoder is the Nth row decoder, a row-level text detection result is obtained according to the second row mask feature.

[0173] Furthermore, if the current row decoder is not the Nth row decoder, return to step 201 and re-execute steps 201 to 206 until the second row mask feature of the Nth row decoder is obtained, and the row-level text detection result is obtained based on the second mask feature of the Nth row decoder.

[0174] Furthermore, the second row of query features is converted into prediction data through the self-attention layer, mask attention layer and feedforward network , according to the forecast data Get the line-level text detection results. Indicates the probability that the row query feature belongs to the text or does not belong to the text, Represents the mask map predicted according to the row query feature, and the mask map has row mask features.

[0175] in, , , Indicates the number of row queries, and Identify the height and width of the image feature respectively, and They represent features of different scales.

[0176] In one embodiment, the second decoder is a word decoder, referring to Figure 8 , the second decoder includes at least one word decoder; the text detection method may include the following steps:

[0177] Step 301, obtaining initial word query features and pre-generated word mask features through the current word decoder.

[0178] For example, the initial word query feature is a word query feature obtained by optimizing the preset word query feature when training the text detection model. The dimension of the preset word query feature can be set according to user needs, for example, the dimension can be set to 256.

[0179] The step 301 of obtaining the pre-generated word mask features includes the following sub-steps:

[0180] Sub-step 3011, if the current word decoder is the first word decoder, convolution processing is performed on the image features to obtain word mask features.

[0181] For example, the image features include image features of multiple scales, and the image features of the largest scale (ie, the image features of the lowest level) are input into the convolutional neural network to obtain word mask features.

[0182] Sub-step 3012, if the current word decoder is the Qth word decoder, the word mask feature obtained according to the word query feature output by the Q-1th word decoder is determined as the word mask feature.

[0183] Among them, Q is greater than 1.

[0184] For example, each word decoder has a corresponding word predictor, and the word query features output by the Q-1th word decoder are input into the corresponding word predictor to obtain a word mask map. The features of the word mask map are the word mask features obtained in this step based on the word query features output by the Q-1th word decoder.

[0185] Further, see Figure 8 , the word decoder outputs the word query feature, and the word query feature is input into the word predictor to obtain the word prediction data, which includes the word mask map, and the feature of the word mask map is the word mask feature. The word mask feature output by the Q-1th word decoder can be obtained according to the self-attention layer, mask attention layer, and feedforward network in the word decoder.

[0186] Step 302, obtaining a word attention feature according to the self-attention mechanism, the initial word query feature, and the first feature.

[0187] Specifically, a self-attention calculation is performed on the initial word query feature, and then the self-attention calculation result and the first feature are multiplied to obtain the word attention feature.

[0188] For example, each line has words, in the word decoder, first initialize The original word query and the first feature are multiplied to obtain the number of words in the word attention feature, which is equal to the product of the number of rows in the image and the words in each row. That is, the number of words in the word attention feature ,in, It is the number of row queries, reflecting the row number information. After being processed by the row encoder, the initial row query feature is updated to the first feature. After the product operation of the original word query and the first feature, the number of words in the word attention feature can be obtained.

[0189] Through the method in this step, the self-attention between words can be decoupled into the self-attention between rows and the self-attention between words in each row. The row query based on the row decoder only needs to calculate Compared with the method of performing self-attention calculation on all word features in the related art, this embodiment reduces the amount of data processing and reduces the calculation cost.

[0190] Furthermore, we can use the formula Get word query features ,in is the row query feature output by the row encoder, is the learnable word query feature for each row. After obtaining the word attention feature, the word query feature is subsequently updated through multiple cascaded mask attention and feedforward networks .

[0191] in, Represents word query features, Indicates the number of word queries, Represents row query characteristics, represents the learnable word attention features for each row, for The transposed matrix of Represents a meta-word query, Indicates the number of rows to query. Indicates the number of channels. , , They represent word query features, row query features, and learnable word query features for each row respectively.

[0192] The word attention feature is obtained by performing self-attention calculation on the initial word query feature and multiplying the calculation result and the first feature. The process of obtaining the word attention feature realizes the decoupling of the self-attention between words. Specifically, it is converted into the self-attention between rows and the self-attention between words in each row. The word attention feature can be obtained by only calculating the self-attention between the initial word query features and then multiplying it by the row query feature. Compared with the method of performing self-attention calculation on all word features in the related art, this embodiment reduces the amount of data processing and reduces the calculation cost.

[0193] Step 303, according to the masked attention mechanism, the word attention feature, the image feature, and the pre-generated word mask feature, obtain the second word query feature and the association relationship between the word and the row.

[0194] Among them, the association relationship between words and rows is Figure 8 The word-row relationship in , the association relationship represents the correspondence between the word and the row to which the word belongs.

[0195] The image features used in the mask attention calculation are the image features of the lower level in the multi-scale image features. For example, the image features include four scales from F1 to F4, and the image scales from F1 to F4 decrease gradually. The word decoders include three, and the three word decoders use the image features of the three scales from F1 to F3.

[0196] For example, the image feature includes multiple image features of different scales, and the scale of the image feature corresponding to the Qth word decoder is greater than the scale of the image feature corresponding to the Q-1th word decoder. Step 303 may include the following steps:

[0197] Sub-step 3031, based on the masked attention mechanism, the word attention features, the image features of the scale corresponding to the current word decoder, and the pre-generated word mask features, obtain the second word query features and the association relationship between the words and the rows.

[0198] In practical applications, compared with lines and paragraphs, the aspect ratio of word text is relatively simple, but it is necessary to extract richer text details through the text detection model to ensure the accuracy of the detection results. For example, multiple cascaded (for example, 3) decoder layers are used to gradually extract word text features from high to low, so as to predict the word-line relationship based on the final word-level detection results. For example, there are 3 row decoders, the first row decoder uses F4 scale image features, the second row decoder uses F3 scale image features, and the third row decoder uses F2 scale image features.

[0199] Step 304: If the current word decoder is the Pth word decoder, the second word query feature is determined as the second feature.

[0200] For example, if the current word decoder is not the Pth word decoder, return to step 301, re-execute steps 301 to 303 until the second word query feature of the Pth word decoder is obtained, and the second word query feature of the Pth word decoder is determined as the second feature.

[0201] For example, after step 304, the following may also be included:

[0202] Step 305: input the second feature into the word predictor in the text detection model to obtain the second word mask feature of the current word decoder.

[0203] For example, the structure of the paragraph predictor is as follows Figure 4As shown. Further, the second feature and the image feature are input into Figure 4 The paragraph predictor shown obtains the second word query feature of the current decoder.

[0204] Step 306: If the current word decoder is the Pth word decoder, a second text detection result is obtained according to the second word mask feature.

[0205] If the current word decoder is not the P-th word decoder, return to step 301 until the second word mask feature of the P-th word decoder is obtained, and the second text detection result is obtained according to the second word mask feature.

[0206] For example, the word decoder includes a self-attention mechanism, a mask attention mechanism and a feedforward network. For example, referring to 6, there can be n word decoders in the text detection model, preferably, n=3.

[0207] The word decoder can be a decoder of Transform structure. The word decoder collects multi-scale image features and obtains word query features corresponding to the word decoder. ,in, Represents word query features, Indicates the number of word queries. For example, the self-attention calculation result of the initial word query feature is obtained, and the self-attention calculation result and the first feature are multiplied to obtain the word query feature.

[0208] Furthermore, the current word decoder collects the mask feature output by the previous word decoder, and obtains the second feature according to the mask feature, inputs the second word query feature into the word predictor corresponding to the current word decoder, and converts the second word query feature into word prediction data through the word predictor. , based on word prediction data Get the second word-level text detection result.

[0209] in, It indicates the probability that the second feature belongs to the text or does not belong to the text, and it indicates the mask map predicted according to the second word query feature.

[0210] Specifically, a self-attention calculation is performed on the initial word query feature, and the calculation result and the first feature are multiplied to obtain a word attention feature. According to the word attention feature, the word mask feature, the image feature, and the mask attention mechanism, a word mask feature is obtained. The word mask feature is input into a second feedforward network corresponding to the word decoder to obtain a second word query feature.

[0211] Among them, the word predictor is Figure 4As shown in FIG. 1 , the second word query feature is input into the multi-layer perceptron, and the output result of the multi-layer perceptron is normalized by the softmax function to obtain the text classification result. The output result of the multi-layer perceptron and the image feature with the largest scale in the multi-scale image feature are subjected to Einstein multiplication to obtain a mask map.

[0212] Through the method of this embodiment, the decoupling of word query and row query in the word decoder can be achieved. Through the word and row query decoupling mechanism of the word decoder, Each meta-word query will predict the words in the row, thereby capturing the word-row relationship. In addition, through the first feature, a large amount of row data can be supervised, so that the initial row query feature can learn a strong word-row relationship.

[0213] For example, we can design a word self-attention decoupling mechanism and a row aggregation mechanism according to the differences and characteristics of text instances at different levels, so as to improve the efficiency of model training and reasoning and reduce the computational cost while ensuring the detection accuracy.

[0214] By using the method of this embodiment, the second feature can be accurately obtained, and the problem of inaccurate word-level text detection results caused by errors in the row prediction results when the word query features are obtained using the row prediction results can be avoided. This improves the accuracy of the word-level text detection results. In addition, based on the method of this embodiment, the association relationship between words and rows can be obtained to obtain the layout relationship between words and rows.

[0215] In one embodiment, the second decoder is a paragraph decoder, referring to Fig. 9 , the second decoder includes at least one paragraph decoder. Fig. 9 , the text detection method may include the following steps:

[0216] Step 401, obtaining pre-generated paragraph query features through the current paragraph decoder.

[0217] For example, step 401 may include the following sub-steps:

[0218] Sub-step 4011, if the current decoder is the first paragraph decoder, the initial paragraph query feature is determined as the pre-generated paragraph query feature.

[0219] The initial paragraph query feature is a paragraph query feature obtained by optimizing the preset paragraph query feature when training the text detection model.

[0220] Sub-step 4012: If the current decoder is the Lth paragraph decoder, the paragraph query feature output by the L-1th paragraph decoder is determined as the pre-generated paragraph query feature.

[0221] For example, the L-1th paragraph decoder obtains the paragraph query feature based on the self-attention layer, the cross-attention layer, and the third feed-forward network.

[0222] Step 402, obtaining paragraph attention features according to the self-attention mechanism and the pre-generated paragraph query features.

[0223] Perform self-attention operation on the pre-generated paragraph query features to obtain paragraph attention features.

[0224] Step 403, according to the cross attention mechanism, the paragraph attention feature and the first feature, obtain the second paragraph query feature and the association relationship between the paragraph and the line.

[0225] The relationship between paragraphs and lines is Fig. 9 The paragraph-line relationship in ,the association relationship represents the correspondence between the line and the paragraph to which the line belongs.

[0226] A cross-attention operation is performed on the paragraph attention features and the image features to obtain the second paragraph query features and the association relationship between paragraphs and lines.

[0227] Furthermore, position embedding is added to the query and key of each self-attention layer and cross-attention layer. In the process of obtaining the paragraph query features output by the Nth paragraph decoder layer, the attention weights learned by the cross-attention of the paragraph decoder can reflect the line-paragraph relationship.

[0228] For example, the paragraph decoder includes a self-attention layer, a cross-attention layer, and a third feed-forward network. The self-attention layer is used to build the relationship between paragraph-level learnable queries, and the final supervised adaptive matching of different paragraphs without repeated predictions, the cross-attention layer is used to aggregate the line-level query features to update the paragraph query, and the third feed-forward network is used to predict the final output second paragraph query features.

[0229] According to the pre-generated paragraph query features and the self-attention mechanism, the paragraph attention features are obtained. According to the paragraph attention features, image features, and the cross-attention mechanism, the cross-attention operation results are obtained. The cross-attention operation results are input into the feedforward network to obtain the second paragraph query features of the current paragraph decoder.

[0230] Step 404: If the current paragraph decoder is S paragraph decoders, the second paragraph query feature is determined as the second feature.

[0231] If the current paragraph decoder is not the Sth paragraph decoder, return to step 401 until the third paragraph query feature of the Sth paragraph decoder is obtained, and the third paragraph query feature is determined as the second feature.

[0232] Preferably, when there are multiple paragraph decoders, the Sth paragraph decoder is the last paragraph decoder therein.

[0233] Step 405: Input the second paragraph query feature of the current decoder into the paragraph predictor in the text detection model to obtain the second paragraph mask feature of the current paragraph decoder.

[0234] For example, refer to Fig. 9 , the paragraph query features output by the decoder are input into the paragraph predictor to obtain paragraph prediction data, the paragraph prediction data includes a paragraph mask map, and the features of the paragraph mask map are the paragraph mask features.

[0235] Step 406: If the current paragraph decoder is the Sth paragraph decoder, a second text detection result is obtained according to the second paragraph mask feature.

[0236] If the current paragraph decoder is not the S paragraph decoder, return to step 401 until the second paragraph mask feature of the Pth paragraph decoder is obtained, and the second text detection result is obtained according to the second paragraph mask feature.

[0237] Example, initialization A paragraph-level learnable query is equivalent to the initial paragraph query feature in the aforementioned embodiment. The query feature is then learned through three cascaded paragraph decoders. Among them, the paragraph decoder is a Transformer structure, and the paragraph decoder includes self-attention, cross-attention and feedforward network. Through the iterative optimization of the three-layer paragraph decoder, the relationship construction of paragraph-level learnable queries and the aggregation of row-level features can be realized.

[0238] For example, refer to Figure 6 There can be n paragraph decoders in the text detection model, preferably, n=3. The paragraph decoder can be a Transform structure decoder. The paragraph decoder collects multi-scale image features and obtains paragraph query features. ,in, represents the paragraph query feature, represents the number of paragraph queries, Indicates characteristics.

[0239] If the current paragraph decoder is S paragraph decoders, the current paragraph decoder obtains the second paragraph query feature as the second feature, and inputs it into the paragraph predictor to convert the paragraph query feature into paragraph prediction data , based on the paragraph prediction data , we can get paragraph-level text detection results.

[0240] in, , . Indicates the probability that the second feature belongs to the text or does not belong to the text, Represents the mask map predicted according to the second feature. and Identifies the height and width of the image feature, respectively.

[0241] In practical applications, the prediction of paragraph-level detection results needs to consider the annotation of paragraph level, while the annotation of paragraph text in real scenes is often rough, which may cause the missing of lines or words in paragraphs. In addition, some paragraphs are irregular in shape and difficult to cover with standard rectangles or quadrilaterals. In this embodiment, instead of directly predicting the position and mask of paragraph text, paragraph detection is performed by row-level prediction aggregation based on the first feature output by the row decoder. The first feature is used to make the paragraph check have stronger prior information, which can avoid inaccurate prediction caused by unclear or irregular annotation.

[0242] According to the above embodiment, the optimized row query (the second row query feature of the Nth row decoder), word query (the second word query feature of the Pth word decoder), and paragraph query (the second paragraph query feature of the Sth paragraph decoder) can be generated respectively through the word decoder, the row query feature, the word query feature, and the paragraph query feature. Finally, the three predictors are used to convert the row query feature, the word query feature, and the paragraph query feature into row-level prediction data. , word-level prediction data and paragraph-level prediction data , and then obtain the text detection results of the corresponding level based on the prediction data of each level. , , They represent the probability that the row query feature, word query feature, and paragraph query feature belong to the text or not. , , They represent the mask images predicted based on the row query features, word query features, and paragraph query features respectively.

[0243] That is, enter the query into the text predictor and the lowest level features in the multi-scale graph features ,The query is mapped through a multi-layer perceptron (MLP) to obtain the first feature. Indicates the number of rows to query , word query quantity , or the number of paragraph queries . represents the query features, It is a specific representation of the query feature.

[0244] According to the above reasoning process, word-level text detection boxes are generated, and then the geometric layout between words and lines is formed based on the word-line relationship learned by the word decoder, and the line-paragraph relationship of the paragraph decoder is used to form the geometric layout between lines and paragraphs, and finally a hierarchical text detection output is obtained.

[0245] According to the method of the aforementioned embodiment, multi-level text can be detected simultaneously, and the geometric layout of the multi-level can be predicted based on the association between words and lines, and the association between paragraphs and lines. For example, the geometric layout between words and lines, and the geometric layout between paragraphs and lines can be predicted, thereby achieving the purpose of more comprehensively capturing text information and improving the accuracy of detection.

[0246] Through the line decoder, word decoder and paragraph decoder of the text detection model in the aforementioned embodiment, the hierarchical relationship can be adaptively learned to improve the coherence and logic of text detection and enhance the understanding of text content. Through the interconnected line decoder, word decoder and paragraph decoder of the text detection model, a hierarchical detection mechanism is implemented, based on which the computational cost can be reduced and the detection efficiency can be improved.

[0247] In addition, in this embodiment, through the collaborative work among the word decoder, line decoder and paragraph decoder, and the adaptive learning of contextual relationships by these decoders, the multi-level text detection results obtained can reflect the contextual relationships between different levels.

[0248] In one embodiment, the image features of this embodiment can be fused with multimodal information to obtain fused feature information with richer information, and the fused feature information can be input into a text detection model to obtain multi-level text detection results to solve the problem that it is difficult to accurately detect text in complex application scenarios relying solely on visual information.

[0249] In one embodiment, referring to Fig.10 This embodiment provides a text detection device 50, including:

[0250] The first acquisition module 501 is used to acquire image features of a target object;

[0251] A second acquisition module 502 is used to input the image feature into a first decoder of the text detection model to obtain a first feature, and obtain a first text detection result according to the first feature;

[0252] The third acquisition module 503 is used to input the first feature into the second decoder of the text detection model to obtain the second feature, and obtain the second text detection result based on the second feature; the second decoder has at least one; wherein the first text detection result and the second text detection result are text detection results of different levels, and the levels of the text detection results include: word level, line level, or paragraph level.

[0253] In an example, the first decoder is a row decoder, the first feature is a first row query feature, and the first text detection result is a row-level text detection result; the second decoder includes a word decoder, the second feature includes the first word query feature, and the second text detection result includes a word-level text detection result; and / or, the second decoder includes a paragraph decoder, the second feature includes the first paragraph query feature, and the second text detection result includes a paragraph text detection result.

[0254] For example, when the first decoder is a row decoder, the second acquisition module 502 includes:

[0255] A first acquisition submodule, used for acquiring a pre-generated row query feature and a pre-generated row mask feature through a current row decoder;

[0256] The second acquisition submodule is used to obtain the first row attention feature according to the self-attention mechanism and the pre-generated row query feature;

[0257] A third acquisition submodule is used to obtain a second row of query features according to the mask attention mechanism, the first row of attention features, the image features, and the pre-generated row mask features;

[0258] The fourth acquisition submodule is used to determine the second row query feature as the first feature if the current row decoder is the Nth row decoder.

[0259] Optionally, the first acquisition submodule may include:

[0260] A first acquisition unit is used to determine the initial row query feature as a pre-generated row query feature if the current row decoder is the first row decoder; the initial row query feature is a row query feature obtained by optimizing the preset row query feature when training the text detection model;

[0261] The second acquisition unit is used to determine the row query feature output by the M-1th row decoder as the pre-generated row query feature if the current row decoder is the Mth row decoder; wherein M is greater than 1.

[0262] Optionally, the first acquisition submodule may include:

[0263] a third acquisition unit, configured to perform convolution processing on the image feature to obtain a pre-generated row mask feature if the current row decoder is the first row decoder;

[0264] The fourth acquisition unit is used to determine the row mask feature obtained according to the row query feature output by the M-1th row decoder as the pre-generated row mask feature if the current row decoder is the Mth row decoder; wherein M is greater than 1.

[0265] Optionally, the third acquisition submodule may include:

[0266] a fifth acquisition unit, configured to perform feature fusion on the first row attention feature and the pre-generated row query feature to obtain a first fused feature, and perform mask attention calculation on the pre-generated row mask feature, the first fused feature, and the image feature to obtain a second row attention feature;

[0267] a sixth acquisition unit, configured to perform feature fusion on the first fused feature and the second row of attention features to obtain a second fused feature, and obtain a third row of attention features according to the second fused feature and the first feedforward network;

[0268] The seventh acquisition unit is used to fuse the third row of attention features and the second fusion features to obtain the second row of query features.

[0269] The optional image features include image features of multiple different scales, the row decoder has image features of corresponding scales, and the scale of the image feature corresponding to the Mth row decoder is greater than the scale of the image feature corresponding to the M-1th row decoder.

[0270] Optionally, when the second decoder is a word decoder, the third acquisition module 503 may include:

[0271] The fifth acquisition submodule is used to acquire the initial word query feature and the pre-generated word mask feature through the current word decoder; the initial word query feature is the word query feature obtained by optimizing the preset word query feature when training the text detection model;

[0272] a sixth acquisition submodule, for obtaining a word attention feature according to the self-attention mechanism, the initial word query feature, and the first feature;

[0273] a seventh acquisition submodule, for obtaining a second word query feature and an association relationship between words and rows according to the mask attention mechanism, the word attention feature, the image feature, and the pre-generated word mask feature;

[0274] The eighth acquisition submodule is used to determine the second word query feature as the second feature if the current word decoder is the Pth word decoder.

[0275] Optionally, the fifth acquisition submodule includes:

[0276] an eighth acquisition unit, configured to perform convolution processing on the image feature to obtain a pre-generated word mask feature if the current word decoder is the first word decoder;

[0277] The ninth acquisition unit is used to determine the word mask feature obtained according to the word query feature output by the Q-1th word decoder as the pre-generated word mask feature if the current word decoder is the Qth word decoder; wherein Q is greater than 1.

[0278] Optionally, the image feature includes multiple image features of different scales, and the scale of the image feature corresponding to the Qth word decoder is greater than the scale of the image feature corresponding to the Q-1th word decoder.

[0279] Optionally, when the second decoder is a paragraph decoder, the third acquisition module 503 may include:

[0280] A ninth acquisition submodule, used for acquiring a pre-generated paragraph query feature through a current paragraph decoder;

[0281] The tenth acquisition submodule is used to obtain paragraph attention features according to the self-attention mechanism and the pre-generated paragraph query features;

[0282] The eleventh acquisition submodule is used to obtain the second paragraph query feature and the association relationship between the paragraph and the line according to the cross attention mechanism, the paragraph attention feature and the image feature;

[0283] The twelfth acquisition submodule is used to determine the second paragraph query feature as the second feature if the current paragraph decoder is S paragraph decoders.

[0284] Optionally, the ninth acquisition submodule includes:

[0285] a tenth acquisition unit, configured to determine the initial paragraph query feature as the second paragraph query feature if the current decoder is the first paragraph decoder; the initial paragraph query feature is a paragraph query feature obtained by optimizing the preset paragraph query feature when training the text detection model;

[0286] The eleventh obtaining unit is used to determine the paragraph query feature output by the L-1th paragraph decoder as the pre-generated paragraph query feature if the current decoder is the Lth paragraph decoder.

[0287] Optionally, the second acquisition module 502 includes:

[0288] A thirteenth acquisition submodule is used to input the first feature and the image feature into a first predictor to obtain a first target mask feature;

[0289] A fourteenth acquisition submodule is used to obtain a first text candidate box according to the first target mask feature;

[0290] The third acquisition module 503 includes:

[0291] A fifteenth acquisition submodule is used to input the second feature and the image feature into a second predictor to obtain a second target mask feature;

[0292] The sixteenth acquisition submodule is used to obtain a first text candidate box according to the second target mask feature.

[0293] Optionally, the thirteenth acquisition submodule includes:

[0294] a twelfth acquisition unit, configured to input the first feature and the image feature into a first predictor to obtain a first text classification score of the first feature and a first mask feature corresponding to the first feature;

[0295] A thirteenth obtaining unit is used to obtain, from the first mask feature, a first target mask feature whose first text classification score is greater than or equal to a first preset score threshold;

[0296] The fifteenth acquisition submodule may include:

[0297] a fourteenth acquisition unit, configured to input the second feature and the image feature into a second predictor to obtain a second text classification score of the second feature and a second mask feature corresponding to the second feature;

[0298] The fifteenth acquisition unit is used to obtain, from the second mask features, a second target mask feature whose second text classification score is greater than or equal to a second preset score threshold.

[0299] Optionally, the twelfth acquisition unit includes:

[0300] A first acquisition subunit is used to input the first feature into the multilayer perceptron in the first predictor to obtain a first query feature;

[0301] A second acquisition subunit, configured to obtain a first mask feature according to the first query feature;

[0302] The fourteenth acquisition unit comprises:

[0303] A third acquisition subunit is used to input the second feature into the multilayer perceptron in the second predictor to obtain a second query feature;

[0304] The fourth acquisition subunit is used to obtain a second mask feature according to the second query feature.

[0305] This embodiment can obtain text detection results of multiple levels at the same time, and the text detection results are independent of the text detection results of the previous level. The text detection results are highly accurate, which solves the problem in related technologies that the prediction results of the next level are predicted based on the prediction results of the previous level, resulting in low accuracy of the prediction results.

[0306] The present disclosure also provides an electronic device, such as Fig.11 As shown, it includes a processor 1101 , a communication interface 1102 , a memory 1103 and a communication bus 1104 , wherein the processor 1101 , the communication interface 1102 , and the memory 1103 communicate with each other via the communication bus 1104 .

[0307] The memory 1103 is used to store computer programs.

[0308] When the processor 1101 is used to execute the program stored in the memory 1103, the following steps are implemented: obtaining image features of the target object; inputting the image features into the first decoder of the text detection model to obtain the first features, and obtaining the first text detection result based on the first features; inputting the first features into the second decoder of the text detection model to obtain the second query features, and obtaining the second text detection result based on the second query features; the second decoder has at least one; wherein the first text detection result and the second text detection result are text detection results of different levels, and the levels of the text detection results include: word level, line level, or paragraph level.

[0309] Among them, the processor 1101 can also implement other steps in the above-mentioned text detection method, which will not be repeated here. The communication bus mentioned in the above-mentioned electronic device can be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus can be divided into an address bus, a data bus, and a control bus. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0310] The communication interface is used for communication between the above electronic device and other devices.

[0311] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0312] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP); it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0313] In another embodiment provided by the present disclosure, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the text detection method in the above embodiment.

[0314] In another embodiment provided by the present disclosure, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the text detection method in the above embodiment.

[0315] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave) mode. The computer-readable storage medium can be any available medium that can be accessed by the computer or a server, data center or other data storage device that contains one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)).

[0316] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0317] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For the embodiments of the device, electronic device, computer-readable storage medium and computer program products containing instructions, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0318] The above are only preferred embodiments of the present disclosure and are not intended to limit the protection scope of the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure are included in the protection scope of the present disclosure.

Claims

1. A text detection method, characterized in that: include: Obtain image features of the target object; Inputting the image feature into a first decoder of a text detection model to obtain a first feature, and obtaining a first text detection result based on the first feature; Inputting the first feature into a second decoder of the text detection model to obtain a second feature, and obtaining a second text detection result based on the second feature; The second decoder has at least one; The first text detection result and the second text detection result are text detection results at different levels, and the levels of the text detection results include: word level, line level, or paragraph level; In the case where the second decoder includes a word decoder; inputting the first feature into the second decoder of the text detection model to obtain a second feature, including: Acquire initial word query features and pre-generated word mask features through the current word decoder; the initial word query features are word query features obtained by optimizing preset word query features when training the text detection model; Obtaining a word attention feature according to the self-attention mechanism, the initial word query feature, and the first feature; Obtaining a second word query feature and an association relationship between a word and a row according to the mask attention mechanism, the word attention feature, the image feature, and the pre-generated word mask feature; If the current word decoder is the Pth word decoder, the second word query feature is determined as the second feature.

2. The method according to claim 1, characterized in that The first decoder is a row decoder, the first feature is a first row query feature, and the first text detection result is a row-level text detection result; The second decoder comprises a word decoder, the second feature comprises a first word query feature, and the second text detection result comprises a word-level text detection result; and / or, The second decoder comprises a paragraph decoder, the second feature comprises a first paragraph query feature, and the second text detection result comprises a paragraph-level text detection result.

3. The method according to claim 1, characterized in that In case the first decoder is a row decoder; Inputting the image feature into a first decoder of a text detection model to obtain a first feature, including: Obtaining pre-generated row query features and pre-generated row mask features through the current row decoder; According to the self-attention mechanism and the pre-generated row query features, the first row of attention features is obtained; Obtaining a second row of query features according to the mask attention mechanism, the first row of attention features, the image features, and the pre-generated row mask features; If the current row decoder is the Nth row decoder, the second row query feature is determined as the first feature.

4. The method according to claim 3, characterized in that Get pre-generated row query features through the current decoder, including: If the current row decoder is the first row decoder, the initial row query feature is determined as a pre-generated row query feature; the initial row query feature is a row query feature obtained by optimizing the preset row query feature when training the text detection model; If the current row decoder is the Mth row decoder, the row query feature output by the M-1th row decoder is determined as the pre-generated row query feature; wherein M is greater than 1.

5. The method according to claim 3, characterized in that: Get the pre-generated row mask features through the current decoder, including: If the current row decoder is the first row decoder, convolution processing is performed on the image feature to obtain a pre-generated row mask feature; If the current row decoder is the Mth row decoder, the row mask feature obtained according to the row query feature output by the M-1th row decoder is determined as the pre-generated row mask feature; wherein M is greater than 1.

6. The method according to claim 3, characterized in that According to the mask attention mechanism, the first row attention feature, the image feature, and the pre-generated row mask feature, a second row query feature is obtained, including: Performing feature fusion on the first row attention feature and the pre-generated row query feature to obtain a first fused feature, and performing mask attention calculation on the first fused feature, the image feature, and the pre-generated row mask feature to obtain a second row attention feature; Performing feature fusion on the first fused feature and the second row of attention features to obtain a second fused feature, and inputting the second fused feature into the first feedforward network to obtain a third row of attention features; The third row of attention features and the second fusion features are fused to obtain the second row of query features.

7. The method according to claim 3, characterized in that The image features include multiple image features of different scales, the row decoders have image features of corresponding scales, and the scale of the image features corresponding to the Mth row decoder is greater than the scale of the image features corresponding to the M-1th row decoder.

8. The method according to claim 1, characterized in that Get the pre-generated word mask features through the current word decoder, including: If the current word decoder is the first word decoder, convolution processing is performed on the image feature to obtain a pre-generated word mask feature; If the current word decoder is the Qth word decoder, the word mask feature obtained according to the word query feature output by the Q-1th word decoder is determined as the pre-generated word mask feature; wherein Q is greater than 1.

9. The method according to claim 1, characterized in that: The image features include multiple image features of different scales, the word decoder has image features of corresponding scales, and the scale of the image features corresponding to the Qth word decoder is greater than the scale of the image features corresponding to the Q-1th word decoder.

10. The method according to claim 1, characterized in that In the case where the second decoder is a paragraph decoder; inputting the first feature into the second decoder of the text detection model to obtain the second feature comprises: Obtain the pre-generated paragraph query features through the current paragraph decoder; According to the self-attention mechanism and the pre-generated paragraph query features, the paragraph attention features are obtained; According to the cross attention mechanism, the paragraph attention feature and the first feature, a second paragraph query feature and an association relationship between the paragraph and the line are obtained; If the current paragraph decoder is S paragraph decoders, the second paragraph query feature is determined as the second feature.

11. The method according to claim 10, characterized in that Get the pre-generated paragraph query features through the current paragraph decoder, including: If the current decoder is the first paragraph decoder, the initial paragraph query feature is determined as a pre-generated paragraph query feature; the initial paragraph query feature is a paragraph query feature obtained by optimizing a preset paragraph query feature when training the text detection model; If the current decoder is the Lth paragraph decoder, the paragraph query feature output by the L-1th paragraph decoder is determined as the pre-generated paragraph query feature.

12. The method according to claim 1, characterized in that The first text detection result is a first text candidate frame; and obtaining the first text detection result according to the first feature includes: Inputting the first feature and the image feature into a first predictor to obtain a first target mask feature; Obtaining a first text candidate box according to the first target mask feature; The second text detection result is a second text candidate frame; and obtaining the second text detection result according to the second feature includes: Inputting the second feature and the image feature into a second predictor to obtain a second target mask feature; A second text candidate box is obtained according to the second target mask feature.

13. The method according to claim 12, characterized in that Inputting the first feature and the image feature into a first predictor to obtain a first target mask feature includes: Inputting the first feature and the image feature into the first predictor to obtain a first text classification score of the first feature and a first mask feature corresponding to the first feature; Obtaining, from the first mask features, a first target mask feature whose first text classification score is greater than or equal to a first preset score threshold; Inputting the second feature and the image feature into a second predictor to obtain a second target mask feature, comprising: Inputting the second feature and the image feature into the second predictor to obtain a second text classification score of the second feature and a second mask feature corresponding to the second feature; A second target mask feature having a second text classification score greater than or equal to a second preset score threshold is obtained from the second mask features.

14. The method according to claim 13, characterized in that Inputting the first feature and the image feature into the first predictor to obtain a first text classification score of the first feature and a first mask feature corresponding to the first feature, including: Inputting the first feature into a multilayer perceptron in a first predictor to obtain a first query feature; Obtaining the first mask feature according to the first query feature; Inputting the second feature and the image feature into a predictor to obtain a second text classification score of the second feature and a second mask feature corresponding to the second feature, including: Inputting the second feature into a multilayer perceptron in a second predictor to obtain a second query feature; The second mask feature is obtained according to the second query feature.

15. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images