Single character detection method, model training method, device, equipment and medium
Patent Information
- Application Number
- CN202380008532.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-31
- Publication Date
- 2024-12-03
AI Technical Summary
Existing single-character detection model training lacks single-character level annotations, resulting in low detection accuracy. Especially in English texts, character adhesion problems are prone to occur, and in Chinese texts, misdetection of Chinese radicals is prone to occur.
The single word detection model is trained through a synthetic text image set and a real scene text image set. The synthetic text image is synthesized from a real scene background image and random text, and has a line text label box and a single word label box. It is trained with a real scene text image set and uses multiple Scale feature fusion residual network uses region scores and affinity scores to predict images for single word detection and segmentation.
It improves the accuracy of the single-character detection model, overcomes the defect of lack of single-character level annotation during training, reduces the problems of character adhesion and misdetection of Chinese character radicals, and enhances the accuracy of single-character detection.
Smart Images

Figure CN119072723A_ABST
Abstract
Description
Single word detection method, model training method, device, equipment and medium Technical Field
[0001] Embodiments of the present invention relate to the field of text detection technology, and in particular to a single-word detection method, a model training method, an apparatus, a device, and a medium. Background Art
[0002] Current text detection datasets rarely have annotations at the single-word level (for Chinese, a single word refers to a single Chinese character; for English, a single word refers to a single letter or character). This results in a lack of training samples for the training of single-word detection models, and the trained single-word detection models are not very accurate.
[0003] There is a line text detection algorithm in the related methods. This algorithm can locate single-word areas and link the detected words into text. However, since the ultimate goal is line text detection, the accuracy requirements for single-word positioning are not high. For English text, it is easy to have problems with detected character adhesion, and for Chinese text, it is easy to have problems with misdetection of Chinese character radicals.
[0004] Summary of the Invention
[0005] Embodiments of the present invention provide a single-word detection method, a model training method, an apparatus, a device and a medium, which are used to solve the problem that when training existing single-word detection models, there is a lack of single-word level annotations, resulting in low accuracy of the trained single-word detection models.
[0006] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:
[0007] In a first aspect, an embodiment of the present invention provides a single-word detection method, comprising:
[0008] Get the text image to be detected;
[0009] A single-word detection model is used to perform single-word detection on the text image to be detected, and a single-word detection frame of the text image to be detected is generated, wherein the single-word detection model is trained by a synthetic text image set and a real scene text image set, the synthetic text images in the synthetic text image set are synthesized by a real scene background image and random text, the synthetic text images have a line text annotation frame and a single-word annotation frame, and the real scene text images in the real scene text image set have a line text annotation frame.
[0010] Optionally, the single word detection model adopts a multi-scale feature fusion residual network.
[0011] Optionally, the synthetic text image set includes at least one of the following: a Chinese single-character synthetic text image set and an English single-character synthetic text image set.
[0012] Optionally, the adopting a single word detection model to perform single word detection on the text image to be detected to generate a single word detection frame of the text image to be detected includes:
[0013] Using the single word detection model to perform single word detection on the text image to be detected, and obtaining a region score prediction image and an affinity score prediction image;
[0014] Binarizing the region score prediction image and the affinity score prediction image to obtain a text region mask image and an affinity mask image;
[0015] Determining a single-word segmentation mask image according to the affinity mask image;
[0016] performing a subtraction process on the single-word segmentation mask image and the text region mask image to obtain a plurality of second connected components on the text region mask image;
[0017] Deleting a connected domain that meets a second condition from the plurality of second connected domains to obtain a remaining target second connected domain, where the second condition includes an area condition and / or a center point height condition of the second connected domain;
[0018] The target second connected domain is expanded, and the minimum circumscribed graph of the expanded target second connected domain is used as the predicted single-word annotation box.
[0019] Optionally, determining the single-word partitioning mask image based on the affinity mask image includes: determining multiple first connected domains in the affinity mask image; deleting connected domains that meet a first condition from the multiple first connected domains to obtain remaining target first connected domains, wherein the first condition includes an area condition and / or a center point height condition of the first connected domain; obtaining the maximum value coordinates of the target first connected domain; generating a partitioning strip with a width of a preset number of pixels with the maximum value coordinates as the center to obtain the single-word partitioning mask image including the partitioning strips.
[0020] Optionally, the first condition includes at least one of the following: a ratio of an area of the first connected domain to an area of the affinity mask image is less than a second threshold; and |region_yc-link_h / / 2|>link_h / / 3, where region_yc is the y-axis coordinate of the center point of the first connected domain, and link_h is the height of the affinity mask image;
[0021] and / or
[0022] The second condition includes at least one of the following: the ratio of the area of the second connected component to the area of the text image to be detected is less than a third threshold; and, |region_yc–crop_img_h / / 2|>crop_img_h / / 3, where region_yc is the y-axis coordinate of the center point of the second connected component, and crop_img__h is the height of the text image to be detected; and, max(text_map[region])<min_text_thre, where max(text_map[region]) is the maximum value of the region scores corresponding to the single-character connected components in the region score prediction image, and min_text_thre is the single-character threshold.
[0023] Optionally, before subtracting the single-character division mask image and the text region mask image, it further includes: eroding the text region mask image;
[0024] The subtraction of the single-character division mask image and the text region mask image includes: for each of the second connected components, if the overlapping area between the division bar and the second connected component is greater than a fourth threshold, and the height difference between the center point of the second connected component and the center point of the first connected component in the corresponding affinity mask image is greater than a fifth threshold, it is determined that there are multiple lines of text in the line text annotation box in the region score prediction image; the height of the division bar is defined according to the second connected component of the text region mask image.
[0025] In a second aspect, an embodiment of the present invention provides a training method for a single-character detection model, including:
[0026] Obtain a synthetic text image set, where the synthetic text images in the synthetic text image set are synthesized by real-scene background images and random texts, and the synthetic text images have line text annotation boxes and single-character annotation boxes;
[0027] Use the synthetic text image set to train an initial algorithm network to obtain an intermediate model for single-character detection;
[0028] Use the intermediate model to process a real-scene text image set to obtain pseudo-labels of the real-scene text images in the real-scene image set. The real-scene text images have line text annotation boxes, and the pseudo-labels are single-character annotation boxes;
[0029] Use the synthetic text image set and the real-scene text image set with the pseudo-labels to train the intermediate model to obtain a single-character detection model for single-character detection.
[0030] Optionally, the synthetic text image set includes at least one of the following: a Chinese single-character synthetic text image set and an English single-character synthetic text image set.
[0031] Optionally, obtaining the synthetic text image set includes:
[0032] Select a real scene background image;
[0033] Segmenting the real scene background image to obtain a plurality of segmented regions, wherein the texture and / or color within each segmented region is consistent;
[0034] Performing image depth estimation on the real scene background image to obtain depth information of each segmented area of the real scene background image;
[0035] Screening the segmented regions according to the size, aspect ratio and / or depth information of the segmented regions to obtain candidate regions;
[0036] Randomly selecting the candidate region;
[0037] For each selected candidate area, the following operations are performed until all candidate areas have been processed: rendering randomly selected text according to the color of the candidate area to obtain rendered text; obtaining a minimum circumscribed matrix of the candidate area; obtaining a spatial plane corresponding to the minimum circumscribed matrix according to the depth information of the candidate area; performing a perspective transformation on the candidate area according to the spatial plane, so that the spatial plane of the candidate area is transformed into a target plane parallel to the screen; pasting the rendered text into the transformed candidate area; and performing an inverse perspective transformation on the transformed candidate area.
[0038] All processed candidate regions are mapped back to the real scene background image to obtain a synthetic text image, and line text annotation boxes and single-word annotation boxes in the synthetic text image are determined.
[0039] Optionally, the initial algorithm network, the intermediate model and the single-word detection model adopt a multi-scale feature fusion residual network.
[0040] Optionally, the process of using the intermediate model to process a real scene text image set may further include:
[0041] For a line text annotation box including four vertices in the real scene text image, the following operations are performed:
[0042] Get the height and width of the text annotation box of the row;
[0043] If the line text annotation box is determined to be an annotation box in a first direction based on the height and width of the line text annotation box, a perspective transformation is performed on the screenshot of the area where the line text annotation box is located to obtain a cropped image in a second direction; and the cropped image in the second direction is scaled to obtain a training image adapted to the intermediate model;
[0044] If it is determined that the line text annotation box is a second-direction annotation box based on the height and width of the line text annotation box, the screenshot of the area where the second-direction annotation box is located is directly scaled to obtain a training image adapted to the intermediate model;
[0045] and / or
[0046] For a line text annotation box including N vertices in the real scene text image, perform the following operations, where N is greater than 4:
[0047] Obtaining a minimum bounding matrix of the line text annotation box, and obtaining an area ratio of the line text annotation box to the minimum bounding matrix;
[0048] If the area ratio is less than a first threshold, determining that the line text annotation box is a curved annotation box, obtaining a training image adapted to the intermediate model based on the four vertices of the minimum circumscribed matrix, and setting pixel values of areas outside the line text annotation box of the training image to 0;
[0049] If the area ratio is greater than or equal to the first threshold, determining that the line text annotation box is substantially a rectangular annotation box, and obtaining a training image adapted to the intermediate model based on four vertices of the minimum circumscribed matrix;
[0050] Wherein, obtaining a training image adapted to the intermediate model according to the four vertices of the minimum circumscribed matrix includes:
[0051] Obtain the height and width of the minimum circumscribed matrix;
[0052] If the minimum circumscribed matrix is determined to be a marked frame in a first direction based on the height and width of the minimum circumscribed matrix, a perspective transformation is performed on a screenshot of the area where the minimum circumscribed matrix is located to obtain a cropped image in a second direction; and the cropped image in the second direction is scaled to obtain a training image adapted to the intermediate model.
[0053] If the minimum circumscribed matrix is determined to be a marking frame in the second direction according to the height and width of the minimum circumscribed matrix, the screenshot of the area where the minimum circumscribed matrix is located is directly scaled to obtain a training image adapted to the intermediate model.
[0054] Optionally, if the text line annotation box in the real scene text image is annotated in Chinese, before performing perspective transformation on the screenshot of the area where the text line annotation box or the minimum circumscribed matrix is located, the method further includes:
[0055] Determining whether the text is vertical based on whether the height of the text labeling box or the minimum circumscribed matrix is greater than the product of the width and a preset multiple;
[0056] If the height of the text line annotation box or the minimum bounding matrix is greater than the product of the width and a preset multiple, it is considered to be vertical text, and the width of the text line annotation box or the minimum bounding matrix is expanded by a preset ratio;
[0057] If the height of the text line annotation box or the minimum bounding matrix is less than or equal to the product of the width and a preset multiple, it is considered not to be vertical text, and the height of the text line annotation box or the minimum bounding matrix is expanded by a preset ratio.
[0058] Optionally, the method further includes:
[0059] Generate an area annotation frame between adjacent characters based on the single-character annotation frame in the training image, wherein the area annotation frame between adjacent characters is obtained according to the following method: for two adjacent single-character annotation frames, obtain the diagonal of each of the single-character annotation frames, wherein the diagonal divides the single-character annotation frame into four triangles, and connects the center points of the upper and lower triangles of the two adjacent single-character annotation frames to obtain the area annotation frame between adjacent characters; the training image includes: the synthetic text image and / or the real scene text image with the pseudo label;
[0060] Encoding the single-word annotation frame and the region annotation frame between adjacent characters of the training image using a Gaussian function to obtain a two-dimensional isotropic Gaussian map of the single-word annotation frame and the region annotation frame between adjacent characters;
[0061] Performing perspective transformation on the two-dimensional isotropic Gaussian image of the single-word annotation frame and mapping it into the single-word annotation frame to obtain a first intermediate image;
[0062] Mapping the two-dimensional isotropic Gaussian image of the region labeled frame between adjacent characters into the region labeled frame between adjacent characters to obtain a second intermediate image;
[0063] The first intermediate image and the second intermediate image are processed to output a region score truth image and an affinity score truth image, wherein the region score represents the probability that each pixel in the single-word annotation box is the center point of the word, and the affinity score represents the probability that each pixel in the region annotation box between adjacent words is the center point of the region between adjacent words; the region score truth image and the affinity score truth image are used for training the initial algorithm network or the intermediate model.
[0064] Optionally, the using the intermediate model to process the real scene text image set to obtain pseudo labels of the real scene text images in the real scene image set includes:
[0065] Binarizing the region score prediction image and affinity score prediction image output by the intermediate model to obtain a text region mask image and an affinity mask image;
[0066] Determining a single-word segmentation mask image according to the affinity mask image;
[0067] performing a subtraction process on the single-word segmentation mask image and the text region mask image to obtain a plurality of second connected components on the text region mask image;
[0068] Deleting a connected domain that meets a second condition from the plurality of second connected domains to obtain a remaining target second connected domain, where the second condition includes an area condition and / or a center point height condition of the second connected domain;
[0069] The target second connected domain is expanded, and the minimum circumscribed graph of the expanded target second connected domain is used as the predicted single-word annotation box.
[0070] Optionally, determining the single-word partitioning mask image based on the affinity mask image includes: determining multiple first connected domains in the affinity mask image; deleting connected domains that meet a first condition from the multiple first connected domains to obtain remaining target first connected domains, wherein the first condition includes an area condition and / or a center point height condition of the first connected domain; obtaining the maximum value coordinates of the target first connected domain; generating a partitioning strip with a width of a preset number of pixels with the maximum value coordinates as the center to obtain the single-word partitioning mask image including the partitioning strips.
[0071] Optionally, the first condition includes at least one of the following: a ratio of an area of the first connected domain to an area of the affinity mask image is less than a second threshold; and |region_yc-link_h / / 2|>link_h / / 3, where region_yc is the y-axis coordinate of the center point of the first connected domain, and link_h is the height of the affinity mask image;
[0072] and / or
[0073] The second condition includes at least one of the following: the ratio of the area of the second connected component to the area of the training image is less than a third threshold; and, |region_yc–crop_img_h / / 2|>crop_img_h / / 3, where region_yc is the y-axis coordinate of the center point of the second connected component, and text_h is the height of the text region mask image; and, max(text_map[region])<min_text_thre, where max(text_map[region]) is the maximum value of the region scores corresponding to the single-character connected components in the region score prediction map, and min_text_thre is the single-character threshold.
[0074] Optionally, before subtracting the single-character division mask image and the text region mask image, it further includes: eroding the text region mask image;
[0075] The subtraction of the single-character division mask image and the text region mask image includes: for each of the second connected components, if the overlapping area of the division bar and the second connected component is greater than a fourth threshold, and the height difference between the center point of the second connected component and the center point of the first connected component in the corresponding affinity mask image is greater than a fifth threshold, it is determined that there are multiple lines of text in the line text annotation box in the region score prediction image; the height of the division bar is defined according to the second connected component of the text region mask image.
[0076] Optionally, each pseudo-label corresponds to a confidence level. The training of the intermediate model using the synthetic text image set and the real-scene text image set with the pseudo-labels includes:
[0077] Calculating the loss according to the confidence level of the pseudo-label;
[0078] Adjusting the parameters of the intermediate model according to the loss.
[0079] Optionally, the confidence level is obtained by the following method:
[0080] The confidence level is calculated using the following formula:
[0081] where S conf (w) is the confidence level, l(w) is the number of single characters manually annotated, and l c (w) is the number of single characters predicted;
[0082] If the confidence level is less than a fifth threshold, and the line text annotation box in the training image is a rectangular annotation box, reset the confidence level to the fifth threshold;
[0083] If the confidence level is less than a fifth threshold, and the line text annotation box in the training image is a curved annotation box, the confidence level is reset to 0.
[0084] In a third aspect, an embodiment of the present invention provides a single-word detection device, comprising:
[0085] An acquisition module is used to acquire the text image to be detected;
[0086] A single-word detection module is used to perform single-word detection on the text image to be detected using a single-word detection model, and generate a single-word detection frame for the text image to be detected, wherein the single-word detection model is trained by a synthetic text image set and a real scene text image set, the synthetic text images in the synthetic text image set are synthesized by a real scene background image and random text, the synthetic text images have a line text annotation frame and a single-word annotation frame, and the real scene text images in the real scene text image set have a line text annotation frame.
[0087] In a fourth aspect, an embodiment of the present invention provides a training device for a single-word detection model, comprising:
[0088] A first acquisition module is used to acquire a set of synthetic text images, wherein the synthetic text images in the synthetic text image set are synthesized by a real scene background image and random text, and the synthetic text images have a line text annotation box and a single word annotation box;
[0089] A first training module is used to train an initial algorithm network using the synthetic text image set to obtain an intermediate model for single word detection;
[0090] a pseudo-label acquisition module, configured to process a real scene text image set using the intermediate model to obtain pseudo-labels for the real scene text images in the real scene image set, wherein the real scene text images have line text annotation boxes, and the pseudo-labels are single-word annotation boxes;
[0091] The second training module is used to train the intermediate model using the synthetic text image set and the real scene text image set with the pseudo label to obtain a single word detection model for single word detection.
[0092] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and runnable on the processor, wherein when the program is executed by the processor, the steps of the single-word detection method as described in the first aspect are implemented, or when the program is executed by the processor, the steps of the training method of the single-word detection model as described in the second aspect are implemented.
[0093] In a sixth aspect, an embodiment of the present invention provides a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the single-word detection method described in the first aspect are implemented, or when the computer program is executed by a processor, the steps of the single-word detection model training method described in the second aspect are implemented.
[0094] In an embodiment of the present invention, a synthetic text image with single-word annotations, which is synthesized from a real-scene background image and random text, is used to train a single-word detection model, thereby overcoming the defect of lacking single-word-level annotated text during training. In addition, the training is combined with a set of real-scene text images with line text annotations to improve the accuracy of the single-word detection model. BRIEF DESCRIPTION OF THE DRAWINGS
[0095] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:
[0096] Fig. 1 is a schematic flow diagram of a training method for a single-character detection model according to an embodiment of the present invention;
[0097] FIG2 is a schematic diagram of an improved CRAFT network according to an embodiment of the present invention;
[0098] 3 is a schematic diagram of a method for expanding the height or width of a line text annotation box or a minimum circumscribed matrix according to an embodiment of the present invention;
[0099] 4 is a schematic diagram of a method for converting a single-word annotation frame and an area annotation frame between adjacent words into a region score ground truth image and an affinity score ground truth image according to an embodiment of the present invention;
[0100] FIG5 is a schematic diagram of a method for generating pseudo labels according to an embodiment of the present invention;
[0101] FIG6 is a schematic flow diagram of a single-word detection method according to an embodiment of the present invention;
[0102] FIG7 is a schematic structural diagram of a training device for a single-word detection model according to an embodiment of the present invention;
[0103] FIG8 is a schematic structural diagram of a single-word detection device according to an embodiment of the present invention;
[0104] FIG9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0105] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0106] Referring to FIG1 , the present invention provides a method for training a single-word detection model, comprising:
[0107] Step 11: obtaining a synthetic text image set, wherein the synthetic text images in the synthetic text image set are synthesized by a real scene background image and random text, and the synthetic text images have a line text annotation box and a single word annotation box;
[0108] The single word in the embodiment of the present invention refers to a single Chinese character for Chinese and a single letter (or character) for English.
[0109] In the embodiment of the present invention, optionally, the synthetic text image set includes at least one of the following: a Chinese single-character synthetic text image set and an English single-character synthetic text image set.
[0110] When the synthetic text image set includes both a Chinese single-character synthetic text image set and an English single-character synthetic text image set, the trained model can detect both single Chinese characters and single English characters.
[0111] In an embodiment of the present invention, optionally, obtaining the synthetic text image set includes generating the synthetic text image set. For example, generating a synthetic text image set of Chinese characters. Of course, in some embodiments, an existing synthetic text image set may also be selected, such as SynthText. SynthText is a large dataset containing approximately 800K synthetic text images of English characters. These synthetic text images are generated by mixing real scene background images with random characters.
[0112] Step 12: using the synthetic text image set to train the initial algorithm network to obtain an intermediate model for single word detection;
[0113] Step 13: Processing the real scene text image set using the intermediate model to obtain pseudo labels of the real scene text images in the real scene text image set, wherein the real scene text images have line text annotation boxes, and the pseudo labels are single word annotation boxes;
[0114] The line text annotation box, ie, the annotation box, marks one or more lines of text on the image.
[0115] The so-called real scene text image set refers to real-life images, such as photographed bills, business cards, street signs, etc.
[0116] Step 14: The intermediate model is trained using the synthetic text image set and the real scene text image set with the pseudo labels to obtain a single word detection model for single word detection.
[0117] Due to the lack of single-word annotation data, in an embodiment of the present invention, single-word level annotations are generated from line-level annotations in a real-scene text image set to fine-tune the intermediate model.
[0118] In an embodiment of the present invention, a synthetic text image with single-word annotations, which is synthesized from a real-scene background image and random text, is used to train a single-word detection model, thereby overcoming the defect of lacking single-word-level annotated text during training. In addition, the training is combined with a set of real-scene text images with line text annotations to improve the accuracy of the single-word detection model.
[0119] In an embodiment of the present invention, optionally, the initial algorithm network, the intermediate model, and the single-word detection model all adopt an improved CRAFT (Character Region Awareness For Text detection) network. The improved CRAFT adopts a multi-scale feature fusion residual network, such as a ResNet50 plus FPN structure, as shown in Figure 2. The improved CRAFT uses ResNet50 to replace the original VGG16_bn as the backbone network. Based on the advantages of residual connections, the network converges faster and has stronger feature extraction capabilities. FPN aggregates multi-scale features and can adapt to scenarios with large font size differences such as business cards and paper documents. After aggregating the multi-scale features output by FPN, two layers of convolution are used to gradually compress the number of model channels (the original model uses four layers for compression. Since the ResNet50 network is deeper, the number of layers is reduced here). Finally, a 2-channel prediction probability map is output. The size of the prediction probability map is W / 2*H / 2*2, where W is the width of the training image and H is the height of the training image. The two prediction probability maps include a region score prediction image and an affinity score prediction image. The region score prediction image represents the region score, which indicates the probability that each pixel is the center of a character; the affinity score prediction image represents the affinity score, which indicates the probability that each pixel is the center point of the region between adjacent characters.
[0120] The intermediate model is also used to convert the region score prediction image and the affinity score prediction image into pseudo labels. The specific conversion method will be described in detail in the subsequent content.
[0121] The following describes a method for generating a synthetic text image in an embodiment of the present invention.
[0122] In the embodiment of the present invention, optionally, obtaining the synthetic text image set includes:
[0123] Step 111: Select a real scene background image;
[0124] For example, the real scene background image currently used for processing may be randomly selected from a plurality of real scene background images.
[0125] The so-called real scene text image set refers to real-shot images.
[0126] Step 112: Segmenting the real scene background image to obtain a plurality of segmented regions, wherein the texture and / or color within each segmented region is consistent;
[0127] In the embodiment of the present invention, the gPb-UCM algorithm may be used to segment the selected real scene background image according to local color and / or texture.
[0128] Step 113: performing image depth estimation on the real scene background image to obtain depth information of each segmented area of the real scene background image;
[0129] In an embodiment of the present invention, a CNN (Convolutional Neural Networks) network may be used to perform image depth estimation on the real scene background image.
[0130] The purpose of image depth estimation is to obtain different planes in the real scene background image for pasting text to achieve a close-to-real effect.
[0131] Step 114: Screening the segmented regions according to the size, aspect ratio and / or depth information of the segmented regions to obtain candidate regions; the candidate regions are used for pasting text;
[0132] In the embodiment of the present invention, optionally, when screening the segmented regions, segmented regions with smaller sizes, segmented regions with extreme aspect ratios, and / or segmented regions that cannot form a plane may be removed.
[0133] The candidate region may be represented by contour points.
[0134] Step 115: Randomly select the candidate area;
[0135] Step 116: For each selected candidate area, perform the following operations until all candidate areas have been processed: render the randomly selected text according to the color of the candidate area to obtain rendered text; obtain the minimum circumscribed matrix of the candidate area; obtain the spatial plane corresponding to the minimum circumscribed matrix according to the depth information of the candidate area; perform perspective transformation on the candidate area according to the spatial plane, so that the spatial plane of the candidate area is transformed into a target plane parallel to the screen; paste the rendered text into the transformed candidate area; perform inverse perspective transformation on the transformed candidate area;
[0136] In an embodiment of the present invention, since the spatial plane where the candidate area is located is not necessarily parallel to the target plane of the screen, it is necessary to use perspective transformation to convert the spatial plane of the candidate area into a target plane parallel to the screen, and then perform text rendering. After the text is rendered, an inverse perspective transformation is performed to transform it back to the original spatial plane. At this time, the rendered text can have a perspective effect, which is close to the real shooting scene.
[0137] In the embodiment of the present invention, the perspective transformation may be: estimating a plane normal vector of the spatial plane of the candidate region, and rotating the plane normal vector so that the spatial plane of the candidate region is converted into a target plane parallel to the screen.
[0138] Step 117: Map all processed candidate regions back to the real scene background image to obtain a synthetic text image, and determine the line text annotation box and the single word annotation box in the synthetic text image.
[0139] Since the pasting position and the pasted text are known during pasting, a line text annotation box and a single word annotation box can be obtained.
[0140] In an embodiment of the present invention, optionally, rendering the randomly selected text according to the color of the candidate area includes: determining the color of the randomly selected text according to the color of the candidate area, for example, the color of the candidate area can be input into a color model to obtain the color of the randomly selected text.
[0141] In the embodiment of the present invention, optionally, the font of the randomly selected text may be randomly generated.
[0142] In the embodiment of the present invention, the generated synthetic text image may include at least one of the following: a Chinese single-character synthetic text image and an English single-character synthetic text image.
[0143] In an embodiment of the present invention, optionally, when the rendered text is pasted onto the target plane, the left and right endpoints of the text (which may be one or more characters, or the left and right endpoints of a line of text if there are multiple characters) are aligned with the left and right sides of the minimum circumscribed matrix, and the upper and lower endpoints of the text are aligned with the upper and lower sides of the minimum circumscribed matrix, that is, the single character is made to occupy as much of the candidate area as possible.
[0144] Through the above method, the embodiment of the present invention can generate multiple synthetic text images with line text annotation boxes and single-word annotation boxes, thereby overcoming the defect of lack of single-word level annotation text when training a single-word detection model.
[0145] The pseudo-label generation process is described in detail below.
[0146] In the embodiment of the present invention, optionally, the processing of the real scene text image set using the intermediate model includes:
[0147] For a line text annotation box including four vertices in the real scene text image, the following operations are performed:
[0148] Get the height and width of the text annotation box of the row;
[0149] If, based on the height and width of the line text annotation box, it is determined that the line text annotation box is an annotation box in a first direction (for example, vertical), a perspective transformation is performed on the screenshot of the area where the line text annotation box is located to obtain a cropped image (crop_image) in a second direction (for example, horizontal); the cropped image in the second direction is scaled to obtain a training image adapted to the intermediate model.
[0150] If the line text annotation box is determined to be an annotation box in the second direction (for example, horizontal) based on the height and width of the line text annotation box, the screenshot of the area where the annotation box in the second direction is located is directly scaled to obtain a training image adapted to the intermediate model.
[0151] In an embodiment of the present invention, the height and width of the line text annotation box can be calculated by first arranging the four coordinate points of the line text annotation box in a clockwise direction (such as upper left, upper right, lower right, and lower left), and then calculating the height and width of the line text annotation box according to the arranged coordinate points.
[0152] In an embodiment of the present invention, optionally, if the height of the line text annotation box is greater than n (n is greater than 1, such as 1.5) * the width of the line text annotation box, the line text annotation box is considered to be a vertical annotation box, and the screenshot of the area where the line text annotation box is located needs to be perspective transformed and converted into a horizontal cropped image for unified processing. Of course, in some embodiments, the horizontal screenshot can also be converted into a vertical cropped image, which is not limited by the present invention.
[0153] In the embodiment of the present invention, optionally, the cropped image in the second direction may be scaled using the following formula: scale_h = min(max(64, round(crop_h / 32)*32), 640) scale_w = min(max(64, round(crop_w / 32)*32), 1280)
[0154] Wherein, scale_h is the height of the scaled training image, scale_w is the width of the scaled training image, crop_h is the height of the cropped image in the second direction or the screenshot in the second direction, crop_w is the width of the cropped image in the second direction or the screenshot in the second direction, 640 is the maximum height of the training image, and 1280 is the maximum width of the training image.
[0155] In the embodiment of the present invention, optionally, the height and width of the training image can be divided evenly by m (such as 32, which can be limited as needed).
[0156] In the above embodiment, the line text annotation box is a rectangular annotation box. In some other embodiments of the present invention, the line text annotation box may not be a rectangular annotation box (the number of vertices is equal to 4), for example, a curved annotation box (the number of vertices is greater than 4). The following describes a method for obtaining a training image adapted to the intermediate model when the line text annotation box is a curved annotation box.
[0157] In the embodiment of the present invention, optionally, the processing of the real scene text image set using the intermediate model includes:
[0158] For a line text annotation box including N vertices in the real scene text image, perform the following operations, where N is greater than 4:
[0159] Obtaining a minimum bounding matrix of the line text annotation box, and obtaining an area ratio of the line text annotation box to the minimum bounding matrix;
[0160] If the area ratio is less than a first threshold (e.g., 0.85), the line text annotation box is determined to be a curved annotation box. In this case, a training image adapted to the intermediate model is obtained based on the four vertices of the minimum circumscribed matrix, and the pixel values of the area outside the line text annotation box of the training image are set to 0 to avoid interference from other text or background.
[0161] If the area ratio is greater than or equal to the first threshold, determining that the line text annotation box is substantially a rectangular annotation box, and obtaining a training image adapted to the intermediate model based on four vertices of the minimum circumscribed matrix;
[0162] Wherein, obtaining a training image adapted to the intermediate model according to the four vertices of the minimum circumscribed matrix includes:
[0163] Obtain the height and width of the minimum circumscribed matrix;
[0164] If the minimum circumscribed matrix is determined to be a marked frame in a first direction based on the height and width of the minimum circumscribed matrix, a perspective transformation is performed on a screenshot of the area where the minimum circumscribed matrix is located to obtain a cropped image in a second direction; and the cropped image in the second direction is scaled to obtain a training image adapted to the intermediate model.
[0165] If the minimum circumscribed matrix is determined to be a marking frame in the second direction according to the height and width of the minimum circumscribed matrix, the screenshot of the area where the minimum circumscribed matrix is located is directly scaled to obtain a training image adapted to the intermediate model.
[0166] In an embodiment of the present invention, for the case where the line text annotation box in the real scene text image set is annotated in Chinese, since the existing line text annotation is manually verified and the character height is accurate, and the width of the single-word annotation box predicted by the intermediate model is relatively not tight enough, the generated Gaussian graph will tend to be flat. Therefore, in the processing of Chinese data, before performing the above-mentioned image cropping (perspective transformation of the screenshot of the area where the line text annotation box or the minimum circumscribed matrix is located), the screenshot of the area where the line text annotation box or the minimum circumscribed matrix is located needs to be additionally expanded in height.
[0167] That is, optionally, referring to FIG3 , if the text line annotation box in the real scene text image is annotated in Chinese, before performing perspective transformation on the screenshot of the area where the text line annotation box or the minimum circumscribed matrix is located, the following steps are further included:
[0168] Determining whether the text is vertical based on whether the height of the text label box or the minimum circumscribed matrix is greater than the product of the width and a preset multiple (a setting greater than 1, such as 1.5);
[0169] If the height of the text line annotation box or the minimum bounding matrix is greater than the product of the width and a preset multiple, it is considered to be vertical text, and the width of the text line annotation box or the minimum bounding matrix is expanded by a preset ratio (e.g., 0.2);
[0170] If the height of the text line annotation box or the minimum bounding matrix is less than or equal to the product of the width and a preset multiple, it is considered not to be vertical text, and the height of the text line annotation box or the minimum bounding matrix is expanded by a preset ratio.
[0171] In an embodiment of the present invention, when using synthetic text images to train the initial algorithm network, and when using synthetic text images and real scene text images to train the intermediate model, it is necessary to first convert the single-word annotation boxes in the training images (synthetic text images and / or real scene text images) and the adjacent text area annotation boxes generated by the single-word annotation boxes into area score true value images and the affinity score true value images, and then compare them with the area score prediction images and the affinity score prediction images generated by the initial algorithm network or the intermediate model, so as to adjust the parameters of the initial algorithm network or the intermediate model.
[0172] The following describes how to convert the single-word annotation box and the region annotation box between adjacent characters into a region score true value image and the affinity score true value image.
[0173] In the embodiment of the present invention, optionally, referring to FIG4 , the method further includes:
[0174] Generate an affinity box between adjacent characters (i.e., the affinity box in FIG4 ) based on the character boxes in the training image, wherein the affinity box between adjacent characters is obtained according to the following method: for two adjacent character boxes, obtain the diagonal of each of the character boxes, wherein the diagonal divides the character box into four triangles, and connects the center points of the upper and lower triangles of the two adjacent character boxes to obtain the affinity box between adjacent characters; the training image includes: the synthetic text image and / or the real scene text image with the pseudo label;
[0175] Encoding the single-word annotation frame and the region annotation frame between adjacent characters of the training image using a Gaussian function to obtain a two-dimensional isotropic Gaussian graph (2D Gaussian) of the single-word annotation frame and the region annotation frame between adjacent characters;
[0176] Performing a perspective transform on the two-dimensional isotropic Gaussian image of the single-word annotation frame and mapping it into the single-word annotation frame to obtain a first intermediate image;
[0177] Mapping the two-dimensional isotropic Gaussian image of the region labeled frame between adjacent characters into the region labeled frame between adjacent characters to obtain a second intermediate image;
[0178] The first intermediate image and the second intermediate image are processed to output a region score truth image and an affinity score truth image, wherein the region score represents the probability that each pixel in the single-word annotation box is the center point of the word, and the affinity score represents the probability that each pixel in the region annotation box between adjacent words is the center point of the region between adjacent words; the region score truth image and the affinity score truth image are used for training the initial algorithm network or the intermediate model.
[0179] In the embodiment of the present invention, optionally, referring to FIG5 , the process of processing the real scene text image set using the intermediate model to obtain pseudo labels of the real scene text images in the real scene image set includes:
[0180] Step 51: Binarize the region score prediction image and affinity score prediction image output by the intermediate model to obtain a text region mask image (see the first row of FIG5 ) and an affinity mask image;
[0181] In the embodiment of the present invention, the region score prediction image text_map may be binarized according to text_thre (region score threshold) to obtain the text region mask image text_mask;
[0182] The affinity score prediction image link_map may be binarized using link_thre (affinity score threshold) to obtain an affinity mask image link_mask.
[0183] text_thre (region score threshold) and link_thre (affinity score threshold) can be set as needed.
[0184] Step 52: Determine a single-word partitioning mask image based on the affinity mask image (see the second row of Figure 5). Optionally, determining the single-word partitioning mask image based on the affinity mask image includes: determining multiple first connected domains in the affinity mask image (white circle areas in the second row of Figure 5); deleting connected domains that meet a first condition from the multiple first connected domains to obtain remaining target first connected domains, where the first condition includes an area condition and / or a center point height condition of the first connected domain; obtaining the maximum value coordinates of the target first connected domain; generating a partitioning bar with a width of a preset number of pixels (the vertical line area in the second row of Figure 5) with the maximum value coordinates as the center to obtain a single-word partitioning mask image including the partitioning bar;
[0185] The single - character division mask image is used to process the single - character adhesion problem on the region score prediction image.
[0186] In an embodiment of the present invention, optionally, the first condition includes at least one of the following: the ratio of the area of the first connected component to the area of the affinity mask image is less than a second threshold (e.g., 0.0025) (too small in area); and, |region_yc - link_h / / 2|>link_h / / 3 (large height difference, likely to be other incomplete characters in adjacent upper and lower lines), where region_yc is the y - axis coordinate of the center point of the first connected component, link_h is the height of the affinity mask image, and " / / " is the integer division symbol.
[0187] Step 53: Subtract the single - character division mask image from the text region mask image (set the positions where the single - character division mask image and the text region mask image have 1 to zero), to obtain multiple second connected components on the text region mask image (see the 3rd row of Figure 5);
[0188] Step 54: Delete the connected components that meet the second condition from the multiple second connected components, to obtain the remaining target second connected components (see the 4th row of Figure 5), where the second condition includes the area condition and / or the center - point height condition of the second connected component;
[0189] In an embodiment of the present invention, optionally, the second condition includes at least one of the following: the ratio of the area of the second connected component to the area of the training image is less than a third threshold (such as 0.003) (a region relatively small in area compared to the training image); and, |region_yc–crop_img_h / / 2|>crop_img_h / / 3 (a region where the deviation of the center - point height from the center - point height of the training image is greater than 1 / 3 of the height of the training image), where region_yc is the y - axis coordinate of the center point of the second connected component, text_h is the height of the text region mask image; and, max(text_map[region])<min_text_thre, where max(text_map[region]) is the maximum value of the region scores corresponding to the single - character connected component in the region score prediction map, and min_text_thre is the single - character threshold.
[0190] Step 55: Dilate the target second connected component, and use the minimum bounding rectangle of the dilated target second connected component as the predicted single - character annotation box (see the 5th row of Figure 5).
[0191] In the embodiment of the present invention, optionally, before performing the subtraction process on the single-word segmentation mask image and the text region mask image, the method further includes: corroding the text region mask image; the corroding may be performed once or twice;
[0192] The subtraction processing of the single-character division mask image and the text area mask image includes: for each second connected domain, if the overlapping area between the division bar and the second connected domain is greater than a fourth threshold (for example, 10 pixels) (a single character is divided), and the height difference between the second connected domain and the center point of the corresponding first connected domain in the affinity mask image is greater than a fifth threshold (for example, 5 pixels) (not in the same row), determining that there are multiple lines of text in the row text annotation box in the area score prediction image; and limiting the height of the division bar according to the second connected domain of the text area mask image.
[0193] In the embodiment of the present invention, optionally, each of the pseudo labels corresponds to a confidence level, and the training of the intermediate model using the synthetic text image set and the real scene text image set with the pseudo labels includes:
[0194] Calculate the loss based on the confidence of the pseudo-label; that is, use it to weight each pixel in the loss calculation to reduce the weight of inaccurate pseudo-labels in the loss;
[0195] Parameters of the intermediate model are adjusted according to the loss.
[0196] In the embodiment of the present invention, optionally, the confidence level is obtained by the following method:
[0197] The confidence level is calculated using the following formula:
[0198] Among them, S conf (w) is the confidence, l(w) is the number of words manually annotated, l c (w) is the number of predicted words;
[0199] If the confidence level is less than a fifth threshold (e.g., 0.5), and the line text annotation box in the training image is a rectangular annotation box (with 4 vertices), the confidence level is reset to the fifth threshold, and the single-word annotation box is evenly divided by the line text annotation box.
[0200] If the confidence level is less than a fifth threshold (eg, 0.5), and the line text annotation box in the training image is a curved annotation box (with more than 4 vertices), the confidence level is reset to 0, i.e., the inaccurately predicted curved annotation area is ignored.
[0201] In the above embodiment, the pseudo-label processing range is expanded to support curved annotations and multi-line text annotation data (multiple lines of text within a single annotation box). Secondly, CRAFT uses the watershed algorithm (an image segmentation algorithm) to segment the region score prediction image output by the intermediate model to obtain single-character frames, but this is prone to English character adhesion. To address this, the embodiment of the present invention uses the affinity score prediction image to segment characters, thereby avoiding English character adhesion.
[0202] Referring to FIG6 , an embodiment of the present invention further provides a single-word detection method, comprising:
[0203] Step 61: Obtain a text image to be detected;
[0204] Step 62: Use a single-word detection model to perform single-word detection on the text image to be detected, and generate a single-word detection frame for the text image to be detected, wherein the single-word detection model is trained by a synthetic text image set and a real scene text image set, the synthetic text images in the synthetic text image set are synthesized by real scene background images and random text, the synthetic text images have a line text annotation frame and a single-word annotation frame, and the real scene text images in the real scene text image set have a line text annotation frame.
[0205] In the present invention, synthetic text images are used to train a single-word detection model, which overcomes the defect of lacking single-word-level annotated text, and is combined with a real-scene text image set for training to improve the accuracy of the single-word detection model.
[0206] Optionally, the single word detection model adopts a multi-scale feature fusion residual network, such as ResNet50 plus FPN structure.
[0207] Optionally, the synthetic text image set includes at least one of the following: a Chinese single-character synthetic text image set and an English single-character synthetic text image set.
[0208] Optionally, the adopting a single word detection model to perform single word detection on the text image to be detected to generate a single word detection frame of the text image to be detected includes:
[0209] Using the single word detection model to perform single word detection on the text image to be detected, and obtaining a region score prediction image and an affinity score prediction image;
[0210] Binarizing the region score prediction image and the affinity score prediction image to obtain a text region mask image and an affinity mask image;
[0211] Determining a single-word segmentation mask image according to the affinity mask image;
[0212] Subtract the single - character division mask image from the text region mask image to obtain multiple second connected components on the text region mask image;
[0213] Delete the connected components that meet the second condition from the multiple second connected components to obtain the remaining target second connected components. The second condition includes the area condition and / or the center point height condition of the second connected component;
[0214] Dilate the target second connected component, and use the minimum bounding rectangle of the dilated target second connected component as the predicted single - character annotation box.
[0215] Optionally, the determining the single - character division mask image according to the affinity mask image includes: determining multiple first connected components in the affinity mask image; deleting the connected components that meet the first condition from the multiple first connected components to obtain the remaining target first connected components. The first condition includes the area condition and / or the center point height condition of the first connected component; obtaining the maximum - value coordinates of the target first connected component; generating a division bar with a width of a preset number of pixels centered on the maximum - value coordinates to obtain a single - character division mask image including the division bar.
[0216] Optionally, the first condition includes at least one of the following: the ratio of the area of the first connected component to the area of the affinity mask image is less than a second threshold; and, |region_yc - link_h / / 2|>link_h / / 3, where region_yc is the y - axis coordinate of the center point of the first connected component, and link_h is the height of the affinity mask image;
[0217] and / or
[0218] The second condition includes at least one of the following: the ratio of the area of the second connected component to the area of the text image to be detected is less than a third threshold; and, |region_yc–crop_img_h / / 2|>crop_img_h / / 3, where region_yc is the y - axis coordinate of the center point of the second connected component, and crop_img__h is the height of the text image to be detected; and, max(text_map[region])<min_text_thre, where max(text_map[region]) is the maximum value of the region scores corresponding to the single - character connected component in the region score prediction image, and min_text_thre is the single - character threshold.
[0219] Optionally, before subtracting the single - character division mask image from the text region mask image, it further includes: eroding the text region mask image;
[0220] The subtraction processing of the single-word division mask image and the text area mask image includes: for each second connected domain, if the overlapping area between the division bar and the second connected domain is greater than a fourth threshold, and the height difference between the second connected domain and the center point of the corresponding first connected domain in the affinity mask image is greater than a fifth threshold, determining that there are multiple lines of text in the row text annotation box in the area score prediction image; and limiting the height of the division bar according to the second connected domain of the text area mask image.
[0221] Referring to FIG. 7 , an embodiment of the present invention further provides a training device 70 for a single-word detection model, comprising:
[0222] A first acquisition module 71 is configured to acquire a set of synthetic text images, wherein the synthetic text images in the set are synthesized from a real scene background image and random text, and the synthetic text images have a line text annotation box and a single word annotation box;
[0223] A first training module 72 is configured to train an initial algorithm network using the synthetic text image set to obtain an intermediate model for single-word detection;
[0224] a pseudo-label acquisition module 73 for processing a real scene text image set using the intermediate model to obtain pseudo-labels for real scene text images in the real scene image set, wherein the real scene text images have line text annotation boxes and the pseudo-labels are single-word annotation boxes;
[0225] The second training module 74 is used to train the intermediate model using the synthetic text image set and the real scene text image set with the pseudo labels to obtain a single word detection model for single word detection.
[0226] In the embodiment of the present invention, optionally, the synthetic text image set includes at least one of the following: a Chinese single-character synthetic text image set and an English single-character synthetic text image set.
[0227] Optionally, the first acquisition module 71 is used to select a real scene background image; segment the real scene background image to obtain a plurality of segmented areas, wherein the texture and / or color in each segmented area are consistent; perform image depth estimation on the real scene background image to obtain depth information of each segmented area of the real scene background image; screen the segmented areas according to the size, aspect ratio and / or depth information of the segmented areas to obtain candidate areas; randomly select the candidate areas; perform the following operations on each selected candidate area until all the candidate areas have been processed: according to the color of the candidate area , render the randomly selected text to obtain the rendered text; obtain the minimum circumscribed matrix of the candidate area; obtain the spatial plane corresponding to the minimum circumscribed matrix according to the depth information of the candidate area; perform perspective transformation on the candidate area according to the spatial plane, so that the spatial plane of the candidate area is converted into a target plane parallel to the screen; paste the rendered text into the converted candidate area; perform inverse perspective transformation on the converted candidate area; map all processed candidate areas back to the real scene background image to obtain a synthetic text image, and determine the line text annotation box and the single word annotation box in the synthetic text image.
[0228] Optionally, the initial algorithm network, the intermediate model and the single-word detection model adopt a multi-scale feature fusion residual network, such as a ResNet50 plus FPN structure.
[0229] Optionally, also include:
[0230] An execution module is configured to perform the following operations on a line text annotation box including four vertices in the real scene text image:
[0231] Get the height and width of the text annotation box of the row;
[0232] If the line text annotation box is determined to be an annotation box in a first direction based on the height and width of the line text annotation box, a perspective transformation is performed on the screenshot of the area where the line text annotation box is located to obtain a cropped image in a second direction; and the cropped image in the second direction is scaled to obtain a training image adapted to the intermediate model;
[0233] If it is determined that the line text annotation box is a second-direction annotation box based on the height and width of the line text annotation box, the screenshot of the area where the second-direction annotation box is located is directly scaled to obtain a training image adapted to the intermediate model;
[0234] and / or
[0235] For a line text annotation box including N vertices in the real scene text image, perform the following operations, where N is greater than 4:
[0236] Obtaining a minimum bounding matrix of the line text annotation box, and obtaining an area ratio of the line text annotation box to the minimum bounding matrix;
[0237] If the area ratio is less than a first threshold, determining that the line text annotation box is a curved annotation box, obtaining a training image adapted to the intermediate model based on the four vertices of the minimum circumscribed matrix, and setting pixel values of areas outside the line text annotation box of the training image to 0;
[0238] If the area ratio is greater than or equal to the first threshold, determining that the line text annotation box is substantially a rectangular annotation box, and obtaining a training image adapted to the intermediate model based on four vertices of the minimum circumscribed matrix;
[0239] Wherein, obtaining a training image adapted to the intermediate model according to the four vertices of the minimum circumscribed matrix includes:
[0240] Obtain the height and width of the minimum circumscribed matrix;
[0241] If the minimum circumscribed matrix is determined to be a marked frame in a first direction based on the height and width of the minimum circumscribed matrix, a perspective transformation is performed on a screenshot of the area where the minimum circumscribed matrix is located to obtain a cropped image in a second direction; and the cropped image in the second direction is scaled to obtain a training image adapted to the intermediate model.
[0242] If the minimum circumscribed matrix is determined to be a marking frame in the second direction according to the height and width of the minimum circumscribed matrix, the screenshot of the area where the minimum circumscribed matrix is located is directly scaled to obtain a training image adapted to the intermediate model.
[0243] Optionally, if the text line annotation box in the real scene text image is annotated in Chinese, before performing perspective transformation on the screenshot of the area where the text line annotation box or the minimum circumscribed matrix is located, the method further includes:
[0244] Determining whether the text is vertical based on whether the height of the text labeling box or the minimum circumscribed matrix is greater than the product of the width and a preset multiple;
[0245] If the height of the text line annotation box or the minimum bounding matrix is greater than the product of the width and a preset multiple, it is considered to be vertical text, and the width of the text line annotation box or the minimum bounding matrix is expanded by a preset ratio;
[0246] If the height of the text line annotation box or the minimum bounding matrix is less than or equal to the product of the width and a preset multiple, it is considered not to be vertical text, and the height of the text line annotation box or the minimum bounding matrix is expanded by a preset ratio.
[0247] Optionally, also include:
[0248] The true value generation module is used to generate an area annotation frame between adjacent characters based on a single-character annotation frame in a training image, wherein the area annotation frame between adjacent characters is obtained according to the following method: for two adjacent single-character annotation frames, the diagonal of each single-character annotation frame is obtained, the diagonal divides the single-character annotation frame into four triangles, and the center points of the upper and lower triangles of the two adjacent single-character annotation frames are connected to obtain the area annotation frame between adjacent characters; the training image includes: the synthetic text image and / or the real scene text image with the pseudo label; the single-character annotation frame and the area annotation frame between adjacent characters of the training image are encoded using a Gaussian function to obtain the single-character annotation frame and the adjacent area annotation frame. A two-dimensional isotropic Gaussian map of the area annotation box between characters; performing perspective transformation on the two-dimensional isotropic Gaussian map of the single-character annotation box and mapping it into the single-character annotation box to obtain a first intermediate image; mapping the two-dimensional isotropic Gaussian map of the area annotation box between adjacent characters to the area annotation box between adjacent characters to obtain a second intermediate image; processing the first intermediate image and the second intermediate image to output a region score true value image and an affinity score true value image, the region score representing the probability that each pixel in the single-character annotation box is the center point of the character, and the affinity score representing the probability that each pixel in the area annotation box between adjacent characters is the center point of the area between adjacent characters; the region score true value image and the affinity score true value image are used for training the initial algorithm network or the intermediate model.
[0249] Optionally, the pseudo-label acquisition module 73 is used to binarize the region score prediction image and the affinity score prediction image output by the intermediate model to obtain a text region mask image and an affinity mask image; determine a single-word division mask image based on the affinity mask image; perform subtraction processing on the single-word division mask image and the text region mask image to obtain multiple second connected domains on the text region mask image; delete the connected domains that meet the second condition from the multiple second connected domains to obtain the remaining target second connected domains, wherein the second condition includes the area condition and / or the center point height condition of the second connected domain; expand the target second connected domain, and use the minimum circumscribed graph of the expanded target second connected domain as the predicted single-word annotation box.
[0250] Optionally, determining the single-character division mask image according to the affinity mask image includes: determining multiple first connected regions in the affinity mask image; deleting the connected regions that meet the first condition from the multiple first connected regions to obtain the remaining target first connected regions, where the first condition includes the area condition and / or the center point height condition of the first connected region; obtaining the maximum value coordinates of the target first connected regions; generating a division bar with a width of a preset number of pixels centered on the maximum value coordinates to obtain a single-character division mask image including the division bar.
[0251] Optionally, the first condition includes at least one of the following: the ratio of the area of the first connected region to the area of the affinity mask image is less than a second threshold; and, |region_yc - link_h / / 2| > link_h / / 3, where region_yc is the y-axis coordinate of the center point of the first connected region, and link_h is the height of the affinity mask image;
[0252] and / or
[0253] The second condition includes at least one of the following: the ratio of the area of the second connected region to the area of the training image is less than a third threshold; and, |region_yc – crop_img_h / / 2| > crop_img_h / / 3, where region_yc is the y-axis coordinate of the center point of the second connected region, and text_h is the height of the text region mask image; and, max(text_map[region]) < min_text_thre, where max(text_map[region]) is the maximum value of the region scores corresponding to the single-character connected regions in the region score prediction map, and min_text_thre is the single-character threshold.
[0254] Optionally, before performing the subtraction process on the single-character division mask image and the text region mask image, it further includes: eroding the text region mask image;
[0255] Performing the subtraction process on the single-character division mask image and the text region mask image includes: for each of the second connected regions, if the overlapping area between the division bar and the second connected region is greater than a fourth threshold, and the height difference between the second connected region and the center point of the corresponding first connected region in the affinity mask image is greater than a fifth threshold, it is determined that there are multiple lines of text in the line text annotation box in the region score prediction image; defining the height of the division bar according to the second connected region of the text region mask image.
[0256] Optionally, each of the pseudo labels corresponds to a confidence level, and the first training module 71 is configured to calculate a loss based on the confidence level of the pseudo label; and adjust the parameters of the intermediate model based on the loss.
[0257] Optionally, the confidence level is obtained by:
[0258] The confidence level is calculated using the following formula:
[0259] Among them, S conf (w) is the confidence, l(w) is the number of words manually annotated, l c (w) is the number of predicted words;
[0260] If the confidence level is less than a fifth threshold, and the line text annotation box in the training image is a rectangular annotation box, resetting the confidence level to the fifth threshold;
[0261] If the confidence level is less than a fifth threshold, and the line text annotation box in the training image is a curved annotation box, the confidence level is reset to 0.
[0262] Referring to FIG8 , an embodiment of the present invention further provides a single-word detection device 80, comprising:
[0263] An acquisition module 81 is used to acquire a text image to be detected;
[0264] The single-word detection module 82 is used to perform single-word detection on the text image to be detected using a single-word detection model, and generate a single-word detection frame for the text image to be detected, wherein the single-word detection model is trained by a synthetic text image set and a real scene text image set, the synthetic text images in the synthetic text image set are synthesized by a real scene background image and random text, the synthetic text images have a line text annotation frame and a single-word annotation frame, and the real scene text images in the real scene text image set have a line text annotation frame.
[0265] Optionally, the single word detection model adopts a multi-scale feature fusion residual network, such as ResNet50 plus FPN structure.
[0266] Optionally, the synthetic text image set includes at least one of the following: a Chinese single-character synthetic text image set and an English single-character synthetic text image set.
[0267] Optionally, the single - character detection module 82 is configured to perform single - character detection on the text image to be detected by using the single - character detection model, so as to obtain a region score prediction image and an affinity score prediction image; perform binarization processing on the region score prediction image and the affinity score prediction image to obtain a text region mask image and an affinity mask image; determine a single - character division mask image according to the affinity mask image; perform subtraction processing on the single - character division mask image and the text region mask image to obtain multiple second connected components on the text region mask image; delete the connected components that meet the second condition from the multiple second connected components to obtain the remaining target second connected components, where the second condition includes the area condition and / or the center - point height condition of the second connected component; perform dilation on the target second connected components, and use the minimum circumscribed rectangle of the dilated target second connected components as the predicted single - character annotation box.
[0268] Optionally, the determining the single - character division mask image according to the affinity mask image includes: determining multiple first connected components in the affinity mask image; deleting the connected components that meet the first condition from the multiple first connected components to obtain the remaining target first connected components, where the first condition includes the area condition and / or the center - point height condition of the first connected component; obtaining the maximum - value coordinates of the target first connected components; generating a division bar with a width of a preset number of pixels centered on the maximum - value coordinates to obtain a single - character division mask image including the division bar.
[0269] Optionally, the first condition includes at least one of the following: the ratio of the area of the first connected component to the area of the affinity mask image is less than a second threshold; and, |region_yc - link_h / / 2|>link_h / / 3, where region_yc is the y - axis coordinate of the center point of the first connected component, and link_h is the height of the affinity mask image;
[0270] and / or
[0271] The second condition includes at least one of the following: the ratio of the area of the second connected component to the area of the text image to be detected is less than a third threshold; and, |region_yc–crop_img_h / / 2|>crop_img_h / / 3, where region_yc is the y - axis coordinate of the center point of the second connected component, and crop_img__h is the height of the text image to be detected; and, max(text_map[region])<min_text_thre, where max(text_map[region]) is the maximum value of the region scores corresponding to the single - character connected component in the region score prediction image, and min_text_thre is the single - character threshold.
[0272] Optionally, before performing the subtraction process on the single-word segmentation mask image and the text area mask image, the method further includes: corroding the text area mask image;
[0273] The subtraction processing of the single-word division mask image and the text area mask image includes: for each second connected domain, if the overlapping area between the division bar and the second connected domain is greater than a fourth threshold, and the height difference between the second connected domain and the center point of the corresponding first connected domain in the affinity mask image is greater than a fifth threshold, determining that there are multiple lines of text in the row text annotation box in the area score prediction image; and limiting the height of the division bar according to the second connected domain of the text area mask image.
[0274] Please refer to Figure 9. An embodiment of the present invention further provides an electronic device 90, including a processor 91, a memory 92, and a computer program stored in the memory 92 and executable on the processor 91. When the computer program is executed by the processor 91, the various processes of the above-mentioned single-word detection method or the single-word detection model training method embodiment are implemented, and the same technical effects can be achieved. To avoid repetition, they will not be described here.
[0275] The embodiment of the present invention further provides a non-transient computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned single-word detection method or the training method embodiment of the single-word detection model is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Wherein, the non-transient computer-readable storage medium is such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0276] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.
[0277] Through the description of the above embodiments, those skilled in the art will clearly understand that the above-mentioned embodiments and methods can be implemented by software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, or optical disk) and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present invention.
[0278] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.
Claims
1. A single word detection method, characterized in that: include: Get the text image to be detected; A single-word detection model is used to perform single-word detection on the text image to be detected, and a single-word detection frame of the text image to be detected is generated, wherein the single-word detection model is trained by a synthetic text image set and a real scene text image set, the synthetic text images in the synthetic text image set are synthesized by real scene background images and random texts, the synthetic text images have a line text annotation frame and a single-word annotation frame, and the real scene text images in the real scene text image set have a line text annotation frame.
2. The method according to claim 1, characterized in that The single word detection model adopts a multi-scale feature fusion residual network.
3. The method according to claim 1, characterized in that The synthetic text image set includes at least one of the following: a Chinese single-character synthetic text image set and an English single-character synthetic text image set.
4. The method according to claim 1, characterized in that The method of using a single word detection model to perform single word detection on the text image to be detected to generate a single word detection frame of the text image to be detected includes: Using the single word detection model to perform single word detection on the text image to be detected, and obtaining a region score prediction image and an affinity score prediction image; Binarizing the region score prediction image and the affinity score prediction image to obtain a text region mask image and an affinity mask image; Determine a single-word segmentation mask image according to the affinity mask image; Subtracting the single-word segmentation mask image from the text region mask image to obtain a plurality of second connected domains on the text region mask image; Deleting a connected domain that satisfies a second condition from the plurality of second connected domains to obtain a remaining target second connected domain, wherein the second condition includes an area condition and / or a center point height condition of the second connected domain; The target second connected domain is expanded, and the minimum circumscribed graph of the expanded target second connected domain is used as a predicted single-word annotation box.
5. The method according to claim 4, characterized in that Determining the single-word segmentation mask image according to the affinity mask image comprises: determining a plurality of first connected domains in the affinity mask image; Deleting a connected domain that satisfies a first condition from the multiple first connected domains to obtain a remaining target first connected domain, wherein the first condition includes an area condition and / or a center point height condition of the first connected domain; Obtaining the maximum value coordinates of the first connected domain of the target; A division strip with a width of a preset number of pixels is generated with the maximum value coordinate as the center, and a single-word division mask image including the division strip is obtained.
6. The method according to claim 5, characterized in that The first condition includes at least one of the following: a ratio of an area of the first connected domain to an area of the affinity mask image is less than a second threshold; and |region_yc-link_h / / 2|>link_h / / 3, wherein region_yc is the y-axis coordinate of the center point of the first connected domain, and link_h is the height of the affinity mask image; and / or The second condition includes at least one of the following: the ratio of the area of the second connected component to the area of the text image to be detected is less than a third threshold; and, |region_yc–crop_img_h / / 2|>crop_img_h / / 3, where region_yc is the y-axis coordinate of the center point of the second connected component, and crop_img__h is the height of the text image to be detected; and, max(text_map[region])<min_text_thre, where max(text_map[region]) is the maximum value of the region scores corresponding to the single-character connected components in the region score prediction image, and min_text_thre is the single-character threshold.
7. The method according to claim 5, wherein Before subtracting the single-character division mask image from the text region mask image, it further includes: eroding the text region mask image; The subtraction of the single-character division mask image from the text region mask image includes: for each of the second connected components, if the overlapping area between the division bar and the second connected component is greater than a fourth threshold, and the height difference between the center point of the second connected component and the center point of the first connected component in the corresponding affinity mask image is greater than a fifth threshold, it is determined that there are multiple lines of text in the line text annotation box in the region score prediction image; the height of the division bar is defined according to the second connected component of the text region mask image.
8. A method for training a single word detection model, characterized in that: It includes: Obtain a synthetic text image set, where the synthetic text images in the synthetic text image set are synthesized from real-scene background images and random text, and the synthetic text images have line text annotation boxes and single-character annotation boxes; Use the synthetic text image set to train an initial algorithm network to obtain an intermediate model for single-character detection; Use the intermediate model to process a real-scene text image set to obtain pseudo-labels of the real-scene text images in the real-scene image set. The real-scene text images have line text annotation boxes, and the pseudo-labels are single-character annotation boxes; Use the synthetic text image set and the real-scene text image set with the pseudo-labels to train the intermediate model to obtain a single-character detection model for single-character detection.
9. The method according to claim 8, characterized in that The synthetic text image set includes at least one of the following: a Chinese single-character synthetic text image set and an English single-character synthetic text image set.
10. The method according to claim 8, characterized in that The obtaining of the synthetic text image set includes: Select a real-scene background image; Segment the real-scene background image to obtain multiple segmented regions, where the texture and / or color within each segmented region are consistent; Perform image depth estimation on the real-scene background image to obtain depth information of each segmented region of the real-scene background image; Filter the segmented regions according to the size, aspect ratio, and / or depth information of the segmented regions to obtain candidate regions; Randomly select the candidate regions; For each selected candidate area, the following operations are performed until all the candidate areas have been processed: rendering the randomly selected text according to the color of the candidate area to obtain the rendered text; obtaining the minimum circumscribed matrix of the candidate area; obtaining the spatial plane corresponding to the minimum circumscribed matrix according to the depth information of the candidate area; performing perspective transformation on the candidate area according to the spatial plane, so that the spatial plane of the candidate area is converted into a target plane parallel to the screen; pasting the rendered text into the converted candidate area; performing inverse perspective transformation on the converted candidate area; All processed candidate regions are mapped back to the real scene background image to obtain a synthetic text image, and line text annotation boxes and single-word annotation boxes in the synthetic text image are determined.
11. The method according to claim 8, characterized in that The initial algorithm network, the intermediate model and the single word detection model adopt a multi-scale feature fusion residual network.
12. The method according to claim 8, characterized in that The process of using the intermediate model to process the real scene text image set also includes: For a line text annotation box including four vertices in the real scene text image, the following operations are performed: Get the height and width of the text annotation box of the line; If the line text annotation box is determined to be an annotation box in the first direction according to the height and width of the line text annotation box, a perspective transformation is performed on the screenshot of the area where the line text annotation box is located to obtain a cropped image in the second direction; the cropped image in the second direction is scaled to obtain a training image adapted to the intermediate model; If it is determined that the line text annotation box is an annotation box of the second direction according to the height and width of the line text annotation box, the screenshot of the area where the annotation box of the second direction is located is directly scaled to obtain a training image adapted to the intermediate model; and / or For a line text annotation box including N vertices in the real scene text image, the following operations are performed, where N is greater than 4: Obtaining a minimum circumscribed matrix of the line text annotation box, and obtaining an area ratio between the line text annotation box and the minimum circumscribed matrix; If the area ratio is less than a first threshold, determining that the line text annotation box is a curved annotation box, obtaining a training image adapted to the intermediate model according to four vertices of the minimum circumscribed matrix, and setting the pixel value of the area outside the line text annotation box of the training image to 0; If the area ratio is greater than or equal to the first threshold, it is determined that the line text annotation box is substantially a rectangular annotation box, and a training image adapted to the intermediate model is obtained according to four vertices of the minimum circumscribed matrix; Wherein, obtaining a training image adapted to the intermediate model according to the four vertices of the minimum circumscribed matrix includes: Obtain the height and width of the minimum external matrix; If the minimum circumscribed matrix is determined to be a marked frame in the first direction according to the height and width of the minimum circumscribed matrix, a screenshot of the area where the minimum circumscribed matrix is located is perspective transformed to obtain a cropped image in the second direction; the cropped image in the second direction is scaled to obtain a training image adapted to the intermediate model; If the minimum circumscribed matrix is determined to be a labeling frame of the second direction according to the height and width of the minimum circumscribed matrix, the screenshot of the area where the minimum circumscribed matrix is located is directly scaled to obtain a training image adapted to the intermediate model.
13. The method according to claim 12, characterized in that If the text line annotation box in the real scene text image is annotated with Chinese characters, before perspective transformation is performed on the screenshot of the area where the text line annotation box or the minimum circumscribed matrix is located, the method further includes: Determining whether the text is vertical text based on whether the height of the text labeling box or the minimum circumscribed matrix is greater than the product of the width and a preset multiple; If the height of the text line annotation box or the minimum bounding matrix is greater than the product of the width and the preset multiple, it is considered to be vertical text, and the width of the text line annotation box or the minimum bounding matrix is expanded by a preset ratio; If the height of the line text annotation box or the minimum circumscribed matrix is less than or equal to the product of the width and a preset multiple, it is considered not to be vertical text, and the height of the line text annotation box or the minimum circumscribed matrix is expanded by a preset ratio.
14. The method according to claim 8, characterized in that Also includes: Generate an area annotation frame between adjacent characters according to the single-character annotation frame in the training image, wherein the area annotation frame between adjacent characters is obtained according to the following method: for two adjacent single-character annotation frames, obtain the diagonal of each of the single-character annotation frames, wherein the diagonal divides the single-character annotation frame into four triangles, and connect the center points of the upper and lower triangles of the two adjacent single-character annotation frames to obtain the area annotation frame between adjacent characters; The training images include: the synthetic text images and / or real scene text images with the pseudo labels; Encoding the single-word annotation frame and the region annotation frame between adjacent characters of the training image using a Gaussian function to obtain a two-dimensional isotropic Gaussian graph of the single-word annotation frame and the region annotation frame between adjacent characters; Performing perspective transformation on the two-dimensional isotropic Gaussian image of the single-word annotation frame and mapping it into the single-word annotation frame to obtain a first intermediate image; Mapping the two-dimensional isotropic Gaussian image of the region labeled frame between adjacent characters into the region labeled frame between adjacent characters to obtain a second intermediate image; The first intermediate image and the second intermediate image are processed to output a region score truth image and an affinity score truth image, wherein the region score represents the probability that each pixel in the single-word annotation box is the center point of the word, and the affinity score represents the probability that each pixel in the region annotation box between adjacent words is the center point of the region between adjacent words; the region score truth image and the affinity score truth image are used for training the initial algorithm network or the intermediate model.
15. The method according to claim 8, characterized in that The adopting the intermediate model to process the real scene text image set to obtain pseudo labels of the real scene text images in the real scene image set includes: Binarizing the region score prediction image and the affinity score prediction image output by the intermediate model to obtain a text region mask image and an affinity mask image; Determine a single-word segmentation mask image according to the affinity mask image; Subtract the single-character division mask image from the text region mask image to obtain multiple second connected components on the text region mask image; Delete the connected components that meet the second condition from the multiple second connected components to obtain the remaining target second connected components, where the second condition includes the area condition and / or the center point height condition of the second connected component; Dilate the target second connected components and use the minimum bounding rectangle of the dilated target second connected components as the predicted single-character annotation box.
16. The method according to claim 15, characterized in that The determining the single-character division mask image according to the affinity mask image includes: Determine multiple first connected components in the affinity mask image; Delete the connected components that meet the first condition from the multiple first connected components to obtain the remaining target first connected components, where the first condition includes the area condition and / or the center point height condition of the first connected component; Obtain the maximum value coordinates of the target first connected components; Generate a division bar with a width of a preset number of pixels centered on the maximum value coordinates to obtain a single-character division mask image including the division bar.
17. The method according to claim 16, wherein the first condition includes at least one of the following: the ratio of the area of the first connected component to the area of the affinity mask image is less than a second threshold; and, |region_yc - link_h / / 2| > link_h / / 3, where region_yc is the y-axis coordinate of the center point of the first connected component, and link_h is the height of the affinity mask image; and / or the second condition includes at least one of the following: the ratio of the area of the second connected component to the area of the training image is less than a third threshold; and, |region_yc – crop_img_h / / 2| > crop_img_h / / 3, where region_yc is the y-axis coordinate of the center point of the second connected component, text_h is the height of the text region mask image; and, max(text_map[region]) < min_text_thre, where max(text_map[region]) is the maximum value of the region scores corresponding to the single-character connected components in the region score prediction map, and min_text_thre is the single-character threshold.
18. The method according to claim 15, wherein before the subtraction process of the single-character division mask image and the text region mask image, it further includes: eroding the text region mask image; The subtraction processing of the single-word division mask image and the text area mask image includes: for each of the second connected domains, if the overlapping area between the division bar and the second connected domain is greater than a fourth threshold, and the height difference between the second connected domain and the center point of the corresponding first connected domain in the affinity mask image is greater than a fifth threshold, determining that there are multiple lines of text in the line text annotation box in the area score prediction image; and limiting the height of the division bar according to the second connected domain of the text area mask image.
19. The method according to claim 8, characterized in that Each of the pseudo labels corresponds to a confidence level, and the intermediate model is trained using the synthetic text image set and the real scene text image set with the pseudo labels, including: Calculating a loss based on the confidence of the pseudo-label; Parameters of the intermediate model are adjusted according to the loss.
20. The method of claim 19, wherein: The confidence level is obtained by the following method: The confidence level is calculated using the following formula: Among them, S conf (w) is the confidence, l(w) is the number of words manually annotated, and l c (w) is the number of predicted words; If the confidence level is less than a fifth threshold, and the line text annotation box in the training image is a rectangular annotation box, resetting the confidence level to the fifth threshold; If the confidence level is less than a fifth threshold, and the line text annotation box in the training image is a curved annotation box, the confidence level is reset to 0.
21. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the single-word detection method as described in any one of claims 1 to 7 are implemented, or when the program is executed by the processor, the steps of the training method of the single-word detection model as described in any one of claims 8 to 20 are implemented.
22. A non-transitory computer-readable storage medium, characterized in that The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the training method for the single-word detection model as described in any one of claims 1 to 7 are implemented, or when the computer program is executed by the processor, the steps of the training method for the single-word detection model as described in any one of claims 8 to 20 are implemented.
Citation Information
Cited By
Photographing method and device, related equipment and computer program product
CN120238731A
Space-level Chinese ancient book image restoration method and related equipment
CN120708237A
