Text detection method and device, storage medium, electronic equipment and program product

Abstract

The invention relates to a text detection method and device, a storage medium, electronic equipment and a program product. The accuracy and efficiency of text detection can be improved. The text detection method comprises the steps of obtaining a to-be-detected text image; the to-be-detected text image is input into a pre-trained text detection model, and the text detection model comprises a feature extraction network, a text region detection branch and a text category detection branch; performing feature extraction on the to-be-detected text image through the feature extraction network to obtain a fused feature map; and determining a text region of the to-be-detected text image according to the fusion feature map through the text region detection branch, and determining a text category of the to-be-detected text image according to the fusion feature map, the text region and a preset detection task instruction through the text category detection branch to obtain a text detection result. The text category includes at least one of a handwritten form and a printed form.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a text detection method, device, storage medium, electronic device, and program product. Background Art

[0002] The current informatization process in the education industry is accelerating, and text recognition technology is playing an increasingly important role in K12 education. With the help of text detection technology, the quality and efficiency of teaching can be greatly improved.

[0003] Among related technologies, Optical Character Recognition (OCR) technology based on traditional image processing can detect text in simple styles. However, if the task requires multiple text categories (such as printed text, handwritten text, etc.), multiple processing is required, which is time-consuming and labor-intensive. In addition, adding classification information during detection can easily lead to missed detections, thereby reducing the accuracy and efficiency of text detection. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a text detection method, device, storage medium, electronic device and program product to solve the problems of related technologies.

[0005] In order to achieve the above objectives, in a first aspect, the present disclosure provides a text detection method, the method comprising: Get the text image to be detected; Inputting the text image to be detected into a pre-trained text detection model, wherein the text detection model includes a feature extraction network, a text region detection branch, and a text category detection branch; Performing feature extraction on the text image to be detected through the feature extraction network to obtain a fusion feature map; The text area of the text image to be detected is determined by the text area detection branch according to the fused feature map, and the text category of the text image to be detected is determined by the text category detection branch according to the fused feature map, the text area and the preset detection task instruction to obtain a text detection result, wherein the text category includes at least one of handwriting and printed text.

[0006] Optionally, the feature extraction network includes a backbone network and a feature pyramid network; The step of extracting features from the text image to be detected by the feature extraction network to obtain a fused feature map includes: Performing convolution processing and downsampling processing on the text image to be detected through the backbone network to obtain candidate feature maps of different scales; and The candidate feature maps are fused through the feature pyramid network to obtain the fused feature map.

[0007] Optionally, determining the text region of the to-be-detected text image according to the fused feature map through the text region detection branch includes: Upsampling the fused feature map through the text region detection branch to obtain a target feature map with the same size as the text image to be detected; Generate a probability map and a threshold map of the text area according to the target feature map; Obtaining a binarized image through differentiable binarization processing according to the probability image and the threshold image; A text area in the to-be-detected text image is determined according to the binarized image.

[0008] Optionally, the text category detection branch includes a first text category detection sub-branch and a second text category detection sub-branch; The determining of the text category of the to-be-detected text image by the text category detection branch according to the fused feature map, the text area, and a preset detection task instruction includes: In a case where the preset detection task instruction is characterized as a full text category detection task, inputting the fused feature map and text region information into the first text category detection sub-branch to determine each text category in the text image to be detected; In a case where the preset detection task instruction is characterized as a target text category detection task, the fused feature map and text area information are input into the second text category detection sub-branch to determine the target text category in the text image to be detected.

[0009] Optionally, the training of the text detection model includes: Acquire a model training sample, wherein the model training sample includes a text image and label information corresponding to the text image, wherein the label information includes text area information and text category information; Dividing the model training samples into a training set, a validation set, and a test set, wherein the training set is used for learning model parameters, the validation set is used for model selection and tuning, and the test set is used to evaluate the performance of the final model; The text detection model is trained according to the training set, the validation set and the test set to obtain a trained text detection model.

[0010] Optionally, obtaining a model training sample includes: Constructing an educational font material library, the material library includes foreground materials and background materials, wherein the foreground materials include handwritten fonts and printed fonts, and the background materials include images related to educational scenes; Performing preprocessing based on the material library to obtain an initial text image, wherein the preprocessing includes foreground synthesis and background synthesis; Annotating each of the initial text images to generate corresponding label information; Performing augmentation processing on the initial text image to obtain a target text image, wherein the augmentation processing includes flip enhancement, and brightness, contrast, and color enhancement; According to each of the target text images, a model training sample is obtained.

[0011] In a second aspect, the present disclosure provides a text detection device, the device comprising: An acquisition module is used to acquire the text image to be detected; An input module, configured to input the text image to be detected into a pre-trained text detection model, wherein the text detection model includes a feature extraction network, a text region detection branch, and a text category detection branch; A feature extraction module is used to extract features of the text image to be detected through the feature extraction network to obtain a fused feature map; A detection module is configured to determine the text region of the text image to be detected based on the fused feature map through the text region detection branch, and to determine the text category of the text image to be detected based on the fused feature map, the text region, and a preset detection task instruction through the text category detection branch, so as to obtain a text detection result, wherein the text category includes at least one of handwriting and printed text.

[0012] In a third aspect, the present disclosure provides a non-temporary computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0013] In a fourth aspect, the present disclosure provides an electronic device, comprising: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of any one of the methods in the first aspect.

[0014] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods in the first aspect.

[0015] Through the above technical solution, a text image to be detected is obtained and input into a pre-trained text detection model, wherein the text detection model includes a feature extraction network, a text region detection branch, and a text category detection branch. The feature extraction network extracts features from the text image to be detected to obtain a fused feature map. The text region of the text image to be detected is determined based on the fused feature map by the text region detection branch. The text category of the text image to be detected is determined based on the fused feature map, the text region, and a preset detection task instruction by the text category detection branch to obtain a text detection result. The text category includes at least one of handwritten and printed text. By adopting a branch processing architecture to share feature extraction results, different categories of text can be identified while locating the text region, thereby improving the accuracy and efficiency of text detection.

[0016] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following detailed description, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 The figure is a flowchart of a text detection method according to an exemplary embodiment of the present disclosure.

[0018] Figure 2 3 is a structural diagram of a text visual feature extraction network according to an exemplary embodiment of the present disclosure.

[0019] Figure 3 FIG. 1 is a schematic diagram showing a text detection result according to an exemplary embodiment of the present disclosure.

[0020] Figure 4 FIG. 1 is a schematic diagram showing a text detection result according to an exemplary embodiment of the present disclosure.

[0021] Figure 5 The figure is a schematic diagram of an educational handwriting font according to an exemplary embodiment of the present disclosure.

[0022] Figure 6 The figure is a schematic diagram of an educational printing font according to an exemplary embodiment of the present disclosure.

[0023] Figure 7 The figure is a schematic diagram of an initial version of a text image according to an exemplary embodiment of the present disclosure.

[0024] Figure 8 The figure is a schematic diagram showing a background synthesis effect according to an exemplary embodiment of the present disclosure.

[0025] Figure 9 The figure is a schematic diagram showing the fusion of foreground and background according to an exemplary embodiment of the present disclosure.

[0026] Figure 10 FIG. 4 is a schematic diagram showing an image flip effect according to an exemplary embodiment of the present disclosure.

[0027] Figure 11 The figure is a schematic diagram showing an online enhancement effect of brightness and contrast color according to an exemplary embodiment of the present disclosure.

[0028] Figure 12 It is a block diagram of a text detection device according to an exemplary embodiment of the present disclosure.

[0029] Figure 13 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0031] As mentioned in the background technology, in the related technology, Optical Character Recognition (OCR) technology based on traditional image processing can detect text in simple styles, but if the task requires multiple text categories (such as printed text, handwritten text, etc.), multiple processing is required, which is time-consuming and labor-intensive, and adding classification information during detection can easily lead to missed detections, thereby reducing the accuracy and efficiency of text detection.

[0032] In view of this, embodiments of the present disclosure provide a text detection method, apparatus, storage medium, electronic device, and program product, which can improve the accuracy and efficiency of text detection.

[0033] Figure 1 A text detection method according to an exemplary embodiment of the present disclosure includes the following steps: Step S101, obtaining a text image to be detected; Step S102: inputting the text image to be detected into a pre-trained text detection model, wherein the text detection model includes a feature extraction network, a text region detection branch, and a text category detection branch; Step S103, extracting features from the text image to be detected through a feature extraction network to obtain a fused feature map; Step S104: The text region of the text image to be detected is determined according to the fused feature map by the text region detection branch, and the text category of the text image to be detected is determined according to the fused feature map, the text region, and the preset detection task instructions by the text category detection branch to obtain a text detection result, wherein the text category includes at least one of handwriting and printed text.

[0034] For example, the text image to be input into the text detection model can be preprocessed to improve text recognition accuracy based on actual conditions. This can include resizing the image, converting it to grayscale, binarizing it (converting it to black and white), removing noise, and correcting image tilt. The goal is to reduce noise in the image and highlight the text.

[0035] Through the above method, a text image to be detected is obtained and input into a pre-trained text detection model, wherein the text detection model includes a feature extraction network, a text region detection branch, and a text category detection branch. The feature extraction network extracts features from the text image to be detected to obtain a fused feature map. The text region of the text image to be detected is determined based on the fused feature map by the text region detection branch. The text category of the text image to be detected is determined based on the fused feature map, the text region, and a preset detection task instruction by the text category detection branch to obtain a text detection result. The text category includes at least one of handwritten and printed text. By adopting a branch processing architecture to share feature extraction results, different categories of text can be identified while locating the text region, thereby improving the accuracy and efficiency of text detection.

[0036] In a possible implementation, the feature extraction network includes a backbone network and a feature pyramid network; The feature extraction network is used to extract features from the text image to be detected to obtain a fused feature map, which may include: Performing convolution and downsampling operations on the text image to be detected based on the backbone network to obtain candidate feature maps of different scales; The candidate feature maps are fused based on the feature pyramid network to obtain a fused feature map.

[0037] It's important to understand that neural networks automatically learn useful features from raw data, unlike traditional machine learning methods, which often require manual feature design and selection. Multi-layered structures can learn features at different levels. For example, when processing image data, the bottom layer might learn low-level features like edges and colors, while the upper layers can learn more abstract features like object shape and structure. They can handle complex nonlinear data and adapt to various data distributions. Therefore, the number of network layers can be increased to handle large-scale data training.

[0038] For example, Figure 2 FIG. 1 is a structural diagram of a text visual feature extraction network according to an exemplary embodiment of the present disclosure. Figure 2 As shown in the figure, its essence is similar to a segmentation network, which can be divided into two components: the backbone network backbone and the feature pyramid network FPN.

[0039] 1) Backbone network It should be understood that the backbone network of the text detection network uses an image classification network, and the ResNet50 network can be used.

[0040] For example, take the input image size of [1,3,640, 640] as an example: The input image size is [1,3,640,640] and enters the backbone network. It first undergoes a convolution calculation to reduce the size to 1 / 2 of the original size, and then undergoes four downsampling to output four scales: 1 / 4, 1 / 8, 1 / 16, and 1 / 32.

[0041] 2) Feature Pyramid Network (FPN) It should be understood that the Feature Pyramid Network (FPN) is a common method that uses convolutional networks to efficiently extract features from images at various dimensions. Its primary function is to fuse features at different scales. The input to the FPN is the output of the backbone network. The feature map calculated by the FPN is a quarter of the height and width of the original image.

[0042] For example, the feature fusion process can be as follows: 1 / 32 feature map: [1, N, 20, 20] is convolved and upsampled 8 times to get an image size of [1, 64, 160, 160].

[0043] 1 / 16 feature map: [1, N, 40, 40] plus double upsampling of the 1 / 32 feature map to obtain a new 1 / 16 feature map, which is then convolved and upsampled by 4 times to obtain an image size of [1, 64, 160, 160].

[0044] 1 / 8 feature map: [1, N, 80, 80] plus twice the upsampling of the new 1 / 16 feature map to obtain a new 1 / 8 feature map, which is then convolved and upsampled by 2 times to obtain an image size of [1, 64, 160, 160].

[0045] 1 / 4 feature map: [1, N, 160, 160] plus twice the new 1 / 8 feature map is upsampled to obtain a new 1 / 4 feature map, and then convolved to obtain an image size of [1, 64, 160, 160].

[0046] The fused feature map: [1, 256, 160, 160] is obtained by merging the 1 / 4, 1 / 8, 1 / 16, and 1 / 32 feature maps at the channel level.

[0047] Through the above method, feature extraction can be performed on the morphology of text in educational scenarios, which conforms to the rules of text morphology and helps to improve the accuracy of text detection.

[0048] In a possible implementation, determining the text region of the to-be-detected text image based on the fused feature map may include: Based on the text region detection branch, the fused feature map is upsampled to obtain a target feature map with the same size as the text image to be detected; Generate a probability map and a threshold map of the text area according to the target feature map; Obtaining a binarized image through differentiable binarization processing according to the probability image and the threshold image; A text area in the to-be-detected text image is determined according to the binarized image.

[0049] For example, the text region branch can use the Head network to upsample the fused feature map from the Feature Pyramid Network (FPN) and resize it back to the original image size from 1 / 4 its original size. This ultimately generates a text region probability map and a text region threshold map. These maps are then merged to produce an output of size [1, 3, 640, 640], which is the binarized text region map. This can be used to accurately locate and segment text regions, providing a foundation for subsequent text recognition tasks.

[0050] For example, the detected text region may be converted into a bounding box form (including relevant coordinate information) to generate a preliminary text region detection result.

[0051] It should be understood that when the text area detection branch is training the model, it can generate probability maps and threshold maps based on the annotation boxes. These are target images generated based on real annotated data and can be used as reference standards during training. Among them, the probability map can provide the model with the target probability of whether each pixel belongs to the text area. The threshold map can provide the model with a reference threshold for adaptive binarization. The image output by the model is compared with these target images, and the loss value is calculated, which can help the model learn how to accurately predict text areas and appropriate binarization thresholds. This method enables the model to learn detailed text detection results directly from the input image, improving the accuracy and flexibility of detection.

[0052] For example, a shrinking method can be used to obtain the probability map labels required for algorithm training. After the annotation box is shrunk, the probability value of the covered area is 1, and the probability value of the remaining area is 0. For the threshold map labels, the distance from each location to the annotation box is calculated. The closer the distance, the higher the threshold. Pixels close to the text area (annotated box) have a higher threshold, while pixels far from the text area have a lower threshold. The threshold map labels required for algorithm training can be obtained by expanding the annotation box. This method helps the model learn more precise text boundaries and provides better adaptive binarization capabilities in the subsequent text detection process.

[0053] For example, regarding the calculation of loss values, the probability map can use binary cross entropy loss, and the threshold map can use differentiable binarization loss. The model parameters are updated through back propagation so that the prediction results gradually approach the target image.

[0054] Specifically, the probability map can use a binary cross-entropy loss function, and is single-label, that is, one input sample corresponds to one classification output (1 or 0). For data D(x, y) containing N samples, the BCE loss is calculated as shown in formulas (1) and (2): (1) (2) Among them, l i is the loss corresponding to the i-th sample. w is a hyperparameter. For single-label binary classification, setting w has no effect. The loss of the i-th sample is calculated as shown in formula (3): (3) It should be understood that the probability map needs to be converted into a binary map, where the pixel value is 1 represents the text area. In the BCE loss calculation formula, the value of y can be set to 1 or 0 for the following reasons: In binary classification problems, y = 1 typically represents the positive class (or target class), and y = 0 typically represents the negative class (or background class). y can be interpreted as a probability distribution of the true label, with y = 1 meaning 100% belongs to the positive class, and y = 0 meaning 0% belongs to the positive class (i.e., 100% belongs to the negative class). Using 0 and 1 gives the loss function mathematically sound properties, particularly when it comes to derivation and optimization. When y = 1, the formula simplifies to -log(p), and when y = 0, it simplifies to -log(1-p), making calculations and understanding more intuitive. Furthermore, 0 and 1 provide a symmetrical range, ensuring that positive and negative classes are treated consistently. The output of most classification models is a probability value between 0 and 1, and using 0 and 1 as target values allows for direct comparison with this output.

[0055] Furthermore, in information theory, cross-entropy itself is defined based on a discrete probability distribution, where the probability of each category is either 1 or 0. In practical applications, a threshold (such as 0.5) can be easily set to convert continuous prediction probabilities into discrete category predictions. In the context of text detection, y = 1 may indicate that a pixel or region belongs to text, while y = 0 may indicate that a pixel or region does not belong to text. Through this binary representation, the model can learn to distinguish between text and non-text areas, thereby achieving accurate text detection.

[0056] It should be understood that the final binarization image uses a differentiable binarization loss function. The standard binarization function is discontinuous and its process is non-differentiable, and cannot be optimized as the semantic segmentation network is trained. To solve this problem, a step function that approximates the binarization process can be used, namely differentiable binarization. The process can be shown as follows: (4) Where Bˆ is the output binarized image, T is the adaptive threshold image learned by the network, and k represents the amplification factor, which is usually set to 50.

[0057] It should be understood that the network can learn to generate an optimal threshold map through optimization algorithms such as backpropagation and gradient descent. The learned threshold can be dynamically adjusted based on the characteristics of the input image, allowing the model to better handle variations such as lighting conditions and text styles. Furthermore, the network's learning method can capture more complex patterns and features, and can learn to generate thresholds that maximize text detection performance. The learned thresholds can be adapted to different datasets and application scenarios, and can be achieved by minimizing the overall loss function. During inference, the network simultaneously outputs a probability map P and a threshold map T. The binarization process uses the values in T as personalized thresholds for each pixel.

[0058] This approach enables a flexible, dynamic, and task-optimized binarization process, which is more powerful and effective than methods using fixed thresholds or simple rules. This approach can better handle complex text detection scenarios and improve detection accuracy and robustness.

[0059] It should be understood that the model training process outputs three graphs: a probability graph, a threshold graph, and a binarization graph. Therefore, when calculating the loss function, we also need to combine these three graphs with their corresponding true labels to construct a three-part loss function. The overall loss function is defined as shown in equations (5) and (6): (5) (6) Among them, L is the total loss, L s is the probability map loss, L b is the binarization loss, L t is the threshold graph loss. ɑ and β are weight coefficients, which can be set to 1 and 10 respectively.

[0060] For L s and L b The loss calculation uses BCELoss (binary cross entropy loss function).

[0061] It should be understood that the loss function provides a quantitative metric to measure the gap between the model's predictions and the true labels, objectively reflecting the model's current performance. By minimizing the loss function, the model can gradually improve its predictive ability. Furthermore, by appropriately designing the loss function, it can better handle class imbalance issues, for example by giving higher weight to minority class samples.

[0062] It should be understood that the goal of text detection is to identify regions within an image that contain text. This is essentially a binary classification problem: each pixel or region either belongs to text (a positive sample) or does not (a negative sample). However, text is often concentrated in certain parts of an image rather than evenly distributed, resulting in an imbalance between positive and negative samples. To address this issue, a false positive set strategy can be used during loss calculation. This strategy is a training method used to address sample imbalance and improve model performance. In machine learning, it is primarily used to direct the model's attention to samples that are difficult to correctly classify. The training data can be balanced by adjusting the ratio of positive to negative samples (e.g., 1:3). An appropriate loss function can be used to automatically adjust the weights of different samples. Specifically, during training, sample weights can be dynamically adjusted based on the model's prediction for each sample. Samples with incorrect predictions are weighted higher, while samples with correct predictions are weighted lower. A portion of samples with incorrect predictions is retained for the next round of training; these samples constitute the false positive set.

[0063] In one possible implementation, the text category detection branch includes a first text category detection sub-branch and a second text category detection sub-branch; The text category detection branch determines the text category of the text image to be detected based on the fused feature map, text area, and preset detection task instructions, which may include: In the case where the preset detection task instruction is characterized as a full text category detection task, the fused feature map and the text region information are input into the first text category detection sub-branch to determine each text category in the text image to be detected; When the preset detection task instruction is characterized as a target text category detection task, the fused feature map and text area information are input into the second text category detection sub-branch to determine the target text category in the text image to be detected.

[0064] It should be understood that the text category detection branch uses the cross-entropy loss function used in the aforementioned probability graph, which measures the difference between the predicted probability distribution and the true label distribution. During training, the model attempts to minimize this cross-entropy loss, thereby learning how to accurately classify text regions into the correct category.

[0065] For example, for the first text category detection sub-branch, a multi-class cross-entropy loss is often used. For the second text category detection sub-branch, a binary cross-entropy loss (calculated separately for each class) is often used. By using the cross-entropy loss function, the model can effectively learn these classification tasks, improving overall detection and classification accuracy.

[0066] For example, the text category detection branch can obtain the feature block of the corresponding area of the text area on the fused feature map based on the input text area information and the fused feature map, and perform convolution processing to obtain the category feature map, and generate a category prediction channel based on the category feature map to determine the text category features.

[0067] For example, handwriting can be divided into English handwriting and Chinese handwriting according to actual application conditions. For example, for text images in a word dictation scenario, only English handwriting needs to be detected, which belongs to the target text category detection task, where the target text category is English handwriting.

[0068] For example, we can associate bounding boxes with the text category information at the corresponding location, assign a corresponding text category label to each detection box, and form position-category pairing information. Furthermore, we can generate the final visual text detection result by drawing a bounding box on the image of the text to be detected and annotating the corresponding text category information.

[0069] In one possible implementation, training of a text detection model may include: Obtaining model training samples, where the model training samples include text images and label information corresponding to the text images, where the label information includes text region information and text category information; The model training samples are divided into training set, validation set and test set. The training set is used to learn model parameters, the validation set is used for model selection and tuning, and the test set is used to evaluate the performance of the final model. The text detection model is trained according to the training set, validation set, and test set to obtain a trained text detection model.

[0070] For example, Figure 3 is a schematic diagram of a text detection result according to an exemplary embodiment of the present disclosure. Figure 3 As shown in the figure, in the full text category detection task scenario, the text areas in the image are drawn with bounding boxes according to the text categories, and the corresponding text category information hand_write and print are marked, thereby generating a visual text detection result.

[0071] For example, Figure 4 is another schematic diagram of text detection results according to an exemplary embodiment of the present disclosure, such as Figure 4 As shown in the figure, in the target text category detection task scenario, a bounding box is drawn for the text area of the handwritten text category in the image, and the corresponding text category information hand_write is marked, thereby generating another visual text detection result.

[0072] In one possible implementation, obtaining a model training sample may include: Build an educational font library, which includes foreground and background materials. Foreground materials include handwritten and printed fonts, and background materials include images related to educational scenarios. Perform preprocessing based on the material library to obtain the initial text image. The preprocessing includes foreground synthesis and background synthesis. Annotate each initial text image to generate corresponding label information; Perform augmentation processing on the initial text image to obtain the target text image. The augmentation processing includes flip enhancement, as well as brightness, contrast and color enhancement; According to each target text image, a model training sample is obtained.

[0073] It should be understood that text detection technology is required in many teaching tasks in educational scenarios. However, in educational scenarios, text has various forms of expression, and the conventional photographing and sampling of teaching materials and their post-processing are limited by the photographing angle and the characteristics of the teaching materials themselves, which are prone to physical deformation, as well as students' different writing rules and scribbling habits, which affect the final text recognition effect. At the same time, the industry currently lacks high-quality datasets with character detection box position information in educational scenarios, which restricts the effectiveness of text recognition models in educational scenarios. Therefore, we can construct training datasets to address the problem of the lack of datasets in educational scenarios and the needs of text recognition networks.

[0074] Figure 5 and Figure 6 They are respectively a schematic diagram of an educational handwritten font and a schematic diagram of an educational printed font according to an exemplary embodiment of the present disclosure. For example, a variety of educational fonts can be collected for educational scenarios according to actual conditions, including printed and handwritten fonts.

[0075] Specifically, in the foreground synthesis stage, we first randomly generate a text sequence using chat-gpt and randomly initialize the foreground text color and size. We then initialize an RGBA color channel image, whose size is determined by the length of the text sequence and the font size.

[0076] It's important to understand that initializing an image with RGBA color channels creates a transparent background for a text sequence. In RGBA, R, G, and B represent the red, green, and blue color channels, respectively, and A represents the transparency channel. During initialization, the transparency of all pixels is typically set to 0 (fully transparent). This approach allows for more flexible processing of each text instance and facilitates subsequent compositing and enhancement operations.

[0077] For example, you can use the PIL (Python Imaging Library) image library to draw each character on the foreground image according to its original position. Using this method to get the initial version of the text image can be as follows Figure 7 shown.

[0078] It should be understood that background synthesis is primarily responsible for synthesizing background images.

[0079] For example, the embodiment of the present disclosure adopts three background synthesis modes: real paper photographed background, scanned background, and mixed background.

[0080] Among them, the real paper photo background can be obtained by collecting a variety of real paper backgrounds and adjusting the brightness thereof. The scanned background is a solid color background, which is basically white and gray tones in various color levels, and can be used to simulate scanned text images. The mixed background is mainly to enhance the information expression ability of the real paper photo background and the scanned background, so that the generated data is richer and more diverse. The specific method can be: select any real paper photo background, and generate a Gaussian noise background image that is consistent with its size. Then add the Gaussian noise background image to the real paper photo background image pixel by pixel with a weight of 0.2 to get the final background. For example, the composite effect of the above three backgrounds can be as follows Figure 8 shown.

[0081] It should be understood that before performing online augmentation, the text image generated in the foreground in the RGBA color space and the generated text background image need to be collaged and fused first.

[0082] Among them, ensure that the foreground and background images are of the same size. The fusion operation can include: traversing each pixel position, blending the foreground and background pixels according to the transparency (Alpha channel) of the foreground image, and saving the fused image as the final result. This process uses the Alpha channel in RGBA to control the degree of blending between the foreground text and the background, which can achieve a smooth fusion effect. Using an image processing library (such as PIL or OpenCV) can simplify this operation. For example, the fusion diagram of the foreground and background can be as follows Figure 9 shown.

[0083] It should be understood that the basic version of text recognition image style is still relatively simple, and online augmentation processing can be used, including: flip enhancement, brightness and contrast color enhancement.

[0084] 1) Flip enhancement module: In practical applications, files may appear in a variety of different ways, including various orientations and angles. Using image flipping preprocessing can help the model learn how to deal with these changes, thereby improving its detection performance in various practical situations. In some cases, there may be fewer file samples in certain orientations or angles. Image flipping can generate more of these samples, and image flipping can generate more training samples, thereby helping the model to better learn and understand the various characteristics and changes of files. This is very helpful for improving the generalization ability and detection accuracy of the model. Therefore, adding image flipping preprocessing can not only improve the accuracy of file detection in the education field, but also help the model better adapt to various practical application scenarios. For example, the image flipping effect can be as follows Figure 10 shown.

[0085] 2) Brightness and contrast color It's important to understand that analyzing text styles in educational settings reveals that text images are often affected by lighting, camera phone color, and other factors, which can reduce the accuracy of text detection models to a certain extent. Furthermore, text detection training sets are limited and struggle to cover a sufficiently diverse range of image brightness, contrast, and color.

[0086] Specifically, in the method of introducing brightness and contrast color enhancement, contrast refers to the measurement of different brightness levels between the brightest white and the darkest black in the light and dark areas of an image. There are two ways to adjust the contrast of an image: increase the brightness of the white screen and reduce the brightness of the black screen. Brightness refers to the overall brightness of the foreground and background. When the brightness is increased, the foreground and background are brightened as a whole, and vice versa. Color refers to the color distortion caused by different lighting. The value of one of the three channels can be randomly increased or decreased to simulate color changes under different lighting conditions or camera color deviations. Slight adjustments can produce subtle changes in hue, and larger adjustments can simulate significant lighting changes or color deviations. For example, the online enhancement effect of brightness and contrast colors can be as follows Figure 11 shown.

[0087] In summary, the text detection model in the embodiments of the present disclosure can detect text and identify different text categories, which is of great significance for product improvement and optimization. Specifically, this model method has the following advantages: 1) Improved detection accuracy: This text detection model can identify and distinguish different categories of text. This enables the model to perform more accurate detection when processing complex and diverse text data, and can provide higher detection accuracy than single-category models.

[0088] 2) Enhanced user experience: By identifying different types of text, we can provide users with more refined services. For example, we can process or display text differently based on its category, making it easier for users to find the information they need, thereby improving the user experience.

[0089] 3) Expanded Application Scope: Multi-category text detection models can adapt to more application scenarios and have stronger generalization capabilities. For example, in fields such as education, news, and social media, they can better process and understand texts of various categories, thus meeting the needs of more users.

[0090] Figure 12 A text detection device 200 is shown according to an exemplary embodiment of the present disclosure. Figure 12 As shown, the text detection device 200 includes: An acquisition module 201 is used to acquire a text image to be detected; An input module 202 is configured to input a text image to be detected into a pre-trained text detection model, wherein the text detection model includes a feature extraction network, a text region detection branch, and a text category detection branch; A feature extraction module 203 is used to extract features from the text image to be detected through a feature extraction network to obtain a fused feature map; The detection module 204 is used to determine the text area of the text image to be detected based on the fused feature map through the text area detection branch, and to determine the text category of the text image to be detected based on the fused feature map, the text area and the preset detection task instructions through the text category detection branch to obtain a text detection result, where the text category includes at least one of handwriting and printed text.

[0091] Optionally, the feature extraction network includes a backbone network and a feature pyramid network; The feature extraction module 203 is used to: Through the backbone network, the text image to be detected is convolved and downsampled to obtain candidate feature maps of different scales; and The candidate feature maps are fused through the feature pyramid network to obtain a fused feature map.

[0092] Optionally, the detection module 204 is configured to: The fused feature map is upsampled through the text region detection branch to obtain a target feature map with the same size as the text image to be detected; Generate the probability map and threshold map of the text area based on the target feature map; According to the probability map and the threshold map, a binarized map is obtained through differentiable binarization processing; Determine the text area in the text image to be detected based on the binarized image.

[0093] Optionally, the text category detection branch includes a first text category detection sub-branch and a second text category detection sub-branch; The detection module 204 is used to: In the case where the preset detection task instruction is characterized as a full text category detection task, the fused feature map and the text region information are input into the first text category detection sub-branch to determine each text category in the text image to be detected; When the preset detection task instruction is characterized as a target text category detection task, the fused feature map and text area information are input into the second text category detection sub-branch to determine the target text category in the text image to be detected.

[0094] Optionally, the training of the text detection model includes the following modules: A model training sample acquisition module is used to acquire model training samples, where the model training samples include text images and label information corresponding to the text images, and the label information includes text area information and text category information; The model training sample partitioning module is used to divide the model training samples into training set, validation set and test set. The training set is used to learn model parameters, the validation set is used for model selection and tuning, and the test set is used to evaluate the performance of the final model. The text detection model training module is used to train the text detection model based on the training set, validation set and test set to obtain a trained text detection model.

[0095] Optionally, the model training sample acquisition module is used to: Build an educational font library, which includes foreground and background materials. Foreground materials include handwritten and printed fonts, and background materials include images related to educational scenarios. Perform preprocessing based on the material library to obtain the initial text image. The preprocessing includes foreground synthesis and background synthesis. Annotate each initial text image to generate corresponding label information; Perform augmentation processing on the initial text image to obtain the target text image. The augmentation processing includes flip enhancement, as well as brightness, contrast and color enhancement; According to each target text image, a model training sample is obtained.

[0096] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0097] Figure 13 FIG. 1 is a block diagram of an electronic device 300 according to an exemplary embodiment. Figure 13 As shown, the electronic device 300 may include: a processor 301 , a memory 302 , and may further include one or more of a multimedia component 303 , an input / output (I / O) interface 304 , and a communication component 305 .

[0098] The processor 301 is used to control the overall operation of the electronic device 300 to complete all or part of the steps in the above-mentioned text detection method. The memory 302 is used to store various types of data to support the operation of the electronic device 300. This data may include, for example, instructions for any application or method operating on the electronic device 300, as well as application-related data. The memory 302 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 303 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 302 or sent through the communication component 305. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 304 provides an interface between the processor 301 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 305 is used for wired or wireless communication between the electronic device 300 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 305 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0099] In an exemplary embodiment, the electronic device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-mentioned text detection method.

[0100] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided. When the program instructions are executed by a processor, the steps of the above-mentioned text detection method are implemented. For example, the computer-readable storage medium may be the aforementioned memory 302 including the program instructions. The program instructions may be executed by the processor 301 of the electronic device 300 to perform the above-mentioned text detection method.

[0101] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device, and the computer program has a code portion for performing the above-mentioned text detection method when executed by the programmable device.

[0102] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the scope of protection of the present disclosure.

[0103] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0104] In addition, the various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A text detection method, characterized in that: The method comprises: Get the text image to be detected; Inputting the text image to be detected into a pre-trained text detection model, wherein the text detection model includes a feature extraction network, a text region detection branch, and a text category detection branch; Performing feature extraction on the text image to be detected through the feature extraction network to obtain a fusion feature map; The text area of the text image to be detected is determined by the text area detection branch according to the fused feature map, and the text category of the text image to be detected is determined by the text category detection branch according to the fused feature map, the text area and the preset detection task instruction to obtain a text detection result, wherein the text category includes at least one of handwriting and printed text.

2. The method according to claim 1, characterized in that The feature extraction network includes a backbone network and a feature pyramid network; The step of extracting features from the text image to be detected by the feature extraction network to obtain a fused feature map includes: Performing convolution processing and downsampling processing on the text image to be detected through the backbone network to obtain candidate feature maps of different scales; as well as, The candidate feature maps are fused through the feature pyramid network to obtain the fused feature map.

3. The method according to claim 1, characterized in that The determining the text region of the to-be-detected text image according to the fused feature map by the text region detection branch includes: Upsampling the fused feature map through the text region detection branch to obtain a target feature map with the same size as the text image to be detected; Generate a probability map and a threshold map of the text area according to the target feature map; Obtaining a binarized image through differentiable binarization processing according to the probability image and the threshold image; A text area in the to-be-detected text image is determined according to the binarized image.

4. The method according to claim 1, wherein The text category detection branch includes a first text category detection sub-branch and a second text category detection sub-branch; The determining of the text category of the to-be-detected text image by the text category detection branch according to the fused feature map, the text area, and a preset detection task instruction includes: In a case where the preset detection task instruction is characterized as a full text category detection task, inputting the fused feature map and text region information into the first text category detection sub-branch to determine each text category in the text image to be detected; In a case where the preset detection task instruction is characterized as a target text category detection task, the fused feature map and text area information are input into the second text category detection sub-branch to determine the target text category in the text image to be detected.

5. The method according to claim 1, wherein The training of the text detection model includes: Acquire a model training sample, wherein the model training sample includes a text image and label information corresponding to the text image, wherein the label information includes text area information and text category information; Dividing the model training samples into a training set, a validation set, and a test set, wherein the training set is used for learning model parameters, the validation set is used for model selection and tuning, and the test set is used to evaluate the performance of the final model; The text detection model is trained according to the training set, the validation set and the test set to obtain a trained text detection model.

6. The method according to claim 5, characterized in that The obtaining of model training samples includes: Constructing an educational font material library, the material library includes foreground materials and background materials, wherein the foreground materials include handwritten fonts and printed fonts, and the background materials include images related to educational scenes; Performing preprocessing based on the material library to obtain an initial text image, wherein the preprocessing includes foreground synthesis and background synthesis; Annotating each of the initial text images to generate corresponding label information; Performing augmentation processing on the initial text image to obtain a target text image, wherein the augmentation processing includes flip enhancement, and brightness, contrast, and color enhancement; According to each of the target text images, a model training sample is obtained.

7. A text detection device, characterized in that: The device comprises: An acquisition module is used to acquire the text image to be detected; An input module, configured to input the text image to be detected into a pre-trained text detection model, wherein the text detection model includes a feature extraction network, a text region detection branch, and a text category detection branch; A feature extraction module is used to extract features of the text image to be detected through the feature extraction network to obtain a fused feature map; A detection module is configured to determine the text region of the text image to be detected based on the fused feature map through the text region detection branch, and to determine the text category of the text image to be detected based on the fused feature map, the text region, and a preset detection task instruction through the text category detection branch, so as to obtain a text detection result, wherein the text category includes at least one of handwriting and printed text.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images