Text detection method based on contrast learning for extracting features at a pixel block level
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2024-01-17
- Publication Date
- 2026-07-21
Smart Images
Figure CN118015607B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a text detection method based on contrastive learning to extract features at the pixel block level. Background Technology
[0002] Scene text detection technology refers to the process of automatically detecting and recognizing text regions in images or videos within natural scenes using computer vision techniques. This technology primarily focuses on accurately locating and extracting text information existing within natural scenes from images or videos. Unlike traditional OCR (Optical Character Recognition) technology, scene text detection technology faces a series of challenging problems, including complex backgrounds, varying text styles, irregular shapes, and multi-directional text. Currently, scene text detection technologies in the industry are mainly divided into two categories: regression-based scene text detection and segmentation-based scene text detection.
[0003] Regression-based scene text detection uses regression to extract features for scene text detection. This method can detect horizontal, vertical, and slanted text, but it cannot fit curved text well, and therefore cannot perform scene text detection on curved text.
[0004] Scene text detection based on segmentation achieves text detection by dividing text regions. However, when using this method, small text regions may be blurred or incorrectly segmented, leading to a decrease in detection accuracy. In addition, when text regions exist in complex backgrounds, the complex backgrounds may interfere with the segmentation of text regions, resulting in inaccurate detection of the text regions. Summary of the Invention
[0005] The purpose of this application is to provide a text detection method based on contrastive learning to extract features at the pixel block level, which can effectively distinguish between text information and interference information, accurately extract text features, and thus greatly improve the accuracy of text detection.
[0006] To achieve the above objectives, embodiments of this application provide a text detection method for extracting features at the pixel-level based on contrastive learning, comprising the following steps: inputting a sample image into an initial model to obtain a feature image output by the backbone network of the initial model, and obtaining a text detection result of the sample image output by the initial model based on the feature image; dividing the feature image into regions, constructing a first loss function based on the contrastive learning idea and feature points of different regions of the feature image; performing text segmentation on the sample image according to the text detection result of the sample image, determining the secondary labels corresponding to the segmented text regions, and constructing a second loss function based on the secondary labels; wherein, the secondary labels are fuzzy labels or false positive labels, the fuzzy labels are used to characterize the fuzzy semantic clustering category of the segmented text regions, and the false positive labels are used to characterize the segmented text regions as background interference information; iteratively training the initial model until convergence according to the first loss function and the second loss function to obtain a trained text detection model; inputting the image to be detected into the text detection model to obtain the text detection result of the image to be detected.
[0007] To achieve the above objectives, embodiments of this application also provide a text detection system for extracting features at the pixel-level based on contrastive learning. The system includes a model training module and a model usage module. The model training module includes an acquisition unit, a first loss function construction unit, a second loss function construction unit, and a training execution unit. The acquisition unit acquires sample images, inputs the sample images into an initial model, obtains the feature images output by the backbone network of the initial model, and obtains the text detection results of the sample images output by the initial model based on the feature images. The first loss function construction unit divides the feature images into regions and constructs a first loss function based on the contrastive learning approach and the feature points of different regions of the feature images. The second loss function construction unit is used to perform text segmentation on the sample image based on the text detection result of the sample image, determine the secondary labels corresponding to the segmented text regions, and construct a second loss function based on the secondary labels. The secondary labels are fuzzy labels or false positive labels. The fuzzy labels are used to characterize the fuzzy semantic clustering category of the segmented text regions, and the false positive labels are used to characterize the segmented text regions as background interference information. The training execution unit is used to iteratively train the initial model until convergence based on the first loss function and the second loss function to obtain the trained text detection model. The model uses a module to input the image to be detected into the text detection model to obtain the text detection result of the image to be detected.
[0008] To achieve the above objectives, embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described text detection method for extracting features at the pixel block level based on contrastive learning.
[0009] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described text detection method for extracting features at the pixel block level based on contrastive learning.
[0010] The embodiments of this application propose a text detection method based on contrastive learning to extract features at the pixel-level. At the pixel level, compared to traditional text detection methods, this application utilizes the idea of contrastive learning to divide the text region of a sample image into multiple pairs of different positive and negative samples (different regions). By learning the differences between positive and negative sample pairs, the discriminability of text features is improved. The trained text detection model can accurately extract text features from images for text detection, achieving more precise detection of text boundaries. At the pixel-level, this application can accurately distinguish between real text samples and false positive samples such as building patterns by assigning secondary labels through fuzzy semantic clustering in challenging scenarios such as complex backgrounds and blurred text. It possesses stronger ability to distinguish interference information, effectively improving the robustness and accuracy of text detection. The finally trained text detection model can achieve relatively accurate feature extraction for images containing text of arbitrary shapes, whether curved or tilted, significantly improving the universality and generalization ability of the text detection model. It has significant practical application value and can promote the development of the field of text image processing.
[0011] In some optional embodiments, the process of region segmentation of the feature image, based on the contrastive learning approach and feature points of different regions of the feature image, and constructing a first loss function, includes: expanding the text region marked on the sample image outward once and contracting it inward three times sequentially, obtaining hard background region, easy gap positive region, hard gap positive region, hard positive region, and easy positive region sequentially from the outside in; performing region segmentation on the feature image according to the five regions on the sample image; based on infoNCE loss, mapping feature points of the hard positive region and the easy positive region of the feature image to a similar space, mapping feature points of the easy gap positive region and the hard gap positive region of the feature image to a similar space, and mapping feature points of the hard background region and the easy gap positive region of the feature image to a more distant space, thus constructing the first loss function; wherein, mapping to a similar space minimizes the infoNCE loss, and mapping to a more distant space maximizes the infoNCE loss. The existence of the first loss function strengthens the adversarial learning of positive and negative samples. The division into five regions—hard background region, easy gap positive region, hard gap positive region, hard positive region, and easy positive region—helps the model better distinguish between text regions and background regions, further improving the detection capability of the trained text detection model.
[0012] In some optional embodiments, the offset for each outward expansion and inward contraction is determined by the following formula:
[0013]
[0014] Where A is the area of the labeled text region, L is the perimeter of the labeled text region, r is the preset scaling factor, and D is the offset.
[0015] In some optional embodiments, the infoNCE loss function is represented by the following function:
[0016]
[0017] Where q is the query vector, representing the feature vector of a feature point in the feature map, and k+ Let k be the vector of the sample points in the sample image that match q. i Let be the vector of the i-th sample point in the sample image, K be the total number of sample points, and τ be a preset temperature coefficient.
[0018] In some optional embodiments, the step of performing text segmentation on the sample image based on the text detection results of the sample image, determining the secondary labels corresponding to the segmented text regions, and constructing a second loss function based on the secondary labels includes: performing pixel-level text segmentation on the sample image based on the text detection results of the sample image to obtain the segmented text regions and their corresponding text content; performing fuzzy semantic clustering on the segmented text regions in a preset number of sample images based on the text content corresponding to the segmented text regions, labeling the segmented text regions of the same fuzzy semantic clustering category with fuzzy labels corresponding to their respective fuzzy semantic clustering categories, and labeling the segmented text regions whose corresponding text content is not real text with false positive labels; mapping the fuzzy labels and the false positive labels to a distant space to construct the second loss function. The existence of the second loss function promotes semantic features for feature extraction in the model, plays an auxiliary learning role, and further improves the detection accuracy of the text detection model.
[0019] In some optional embodiments, the sample images are obtained through the following steps: acquiring several images containing text and images not containing text as base material images; performing preprocessing on the base material images, including scaling, cropping, and resolution normalization, and converting the preprocessed images into grayscale images; and using all the converted grayscale images as sample images to generate a training sample set.
[0020] In some optional embodiments, the backbone network adopts a ResNet+FPN structure; the backbone network extracts features from the sample images through four branches to obtain intermediate feature images of size 1 / 4, 1 / 8, 1 / 16, and 1 / 32; the intermediate feature images of size 1 / 8, 1 / 16, and 1 / 32 are upsampled to obtain three intermediate feature images of size 1 / 4; the four intermediate feature images of size 1 / 4 are then concatenated in sequence to obtain the feature image output by the backbone network. Attached Figure Description
[0021] Figure 1 This is a flowchart of a text detection method based on contrastive learning to extract features at the pixel block level, provided in one embodiment of this application;
[0022] Figure 2 This is a flowchart of an embodiment of the present application for dividing a feature image into regions, constructing a first loss function based on the idea of contrastive learning and feature points of different regions of the feature image;
[0023] Figure 3 This is a schematic diagram of five regions in a sample image provided in one embodiment of this application;
[0024] Figure 4 This is a flowchart provided in one embodiment of the present application for performing text segmentation on a sample image based on the text detection results of the sample image, determining the secondary labels corresponding to the segmented text regions, and constructing a second loss function based on the secondary labels;
[0025] Figure 5 This is a schematic diagram of the structure of a text detection system that extracts features at the pixel block level based on contrastive learning, provided in another embodiment of this application;
[0026] Figure 6 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.
[0028] One embodiment of this application proposes a text detection method based on contrastive learning to extract features at the pixel block level, applied to an electronic device. The electronic device can be a terminal or a server; in this embodiment and the following embodiments, the electronic device is described using a server as an example. The implementation details of the text detection method based on contrastive learning to extract features at the pixel block level proposed in this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution.
[0029] The specific process of the text detection method based on contrastive learning for extracting features at the pixel level proposed in this embodiment can be described as follows: Figure 1 As shown, it includes:
[0030] Step 101: Input the sample image into the initial model to obtain the feature image output by the backbone network of the initial model, and obtain the text detection result of the sample image based on the feature image output by the initial model.
[0031] In the specific implementation, the server inputs sample images from the training sample set into the constructed initial model. The initial model mainly consists of a backbone network, a classifier, and an output layer. The sample images are actually input into the backbone network of the initial model, which extracts features from the sample images and outputs feature images. The feature images are then fed into the classifier of the initial model for classification, resulting in a probability map. The output layer of the initial model then outputs the text detection results of the sample images based on the probability map. In this process, the server obtains the feature images output by the backbone network and the text detection results of the sample images finally output by the initial model.
[0032] In one example, before training the text detection model, the server first acquires several images containing text and images without text as base source images. Then, each base source image undergoes preprocessing, including scaling, cropping, and resolution normalization. The preprocessed images have consistent and moderate resolution, facilitating feature extraction from the model's backbone network. After preprocessing all base source images, the server converts them to grayscale images and uses all the converted grayscale images as sample images to generate the training sample set.
[0033] In one example, the initial model's backbone network uses a ResNet+FPN structure. The backbone network extracts features from the input sample image through four branches, resulting in four intermediate feature images: a 1 / 4-inch image, an 1 / 8-inch image, a 1 / 16-inch image, and a 1 / 32-inch image. The server then upsamples these three intermediate feature images, obtaining three more 1 / 4-inch images. Finally, these four 1 / 4-inch images are concatenated sequentially to obtain the feature image output by the backbone network, which has the same size as the sample image.
[0034] Step 102: Divide the feature image into regions and construct the first loss function based on the contrastive learning idea and the feature points of different regions of the feature image.
[0035] In the specific implementation, after the server obtains the feature images corresponding to the sample images output by the backbone network, it can divide the feature images into regions. Based on the contrastive learning approach and the feature points of the different regions divided from the feature images, a first loss function is constructed. The first loss function is the contrastive learning loss, used to improve the discriminability of text features.
[0036] In one example, the server divides the feature image into regions. Based on the contrastive learning approach and the feature points of different regions in the feature image, a first loss function is constructed. This can be achieved through the steps shown in the figure, specifically including:
[0037] Step 1021: Expand the text region marked on the sample image outward once and shrink it inward three times in sequence, from the outside to the inside, to obtain the hard background region, easy gap positive region, hard gap positive region, hard positive region and easy positive region.
[0038] In the specific implementation, the sample image is labeled with text regions. The server expands these labeled text regions outward once and contracts them inward three times, resulting in five regions from the outside in: hard background region, easy gap positive region, hard gap positive region, hard positive region, and easy positive region. These five regions can be... Figure 3 As shown, Figure 3 The bold rectangle in the middle represents the text area marked on the sample image.
[0039] In one example, the server determines the offset by which it expands outward and contracts inward the labeled text region on the sample image each time using the following formula:
[0040]
[0041] In the formula, A is the area of the labeled text region, L is the perimeter of the labeled text region, r is the preset scaling factor, and D is the offset. The server first expands the labeled text region on the sample image outward by D, and then shrinks it inward by D, 2D, and 3D in sequence.
[0042] Step 1022: Divide the feature image into regions based on the five regions on the sample image.
[0043] In the specific implementation, after the server obtains the five regions on the sample image, it divides the feature image into corresponding regions based on the boundaries of these five regions, thereby obtaining the hard background region, easy gap positive region, hard gap positive region, hard positive region, and easy positive region of the feature image.
[0044] Step 1023: Based on infoNCE loss, the feature points of the hard positive region and the easy positive region of the feature image are mapped to a similar space, the feature points of the easy gap positive region and the hard gap positive region of the feature image are mapped to a similar space, and the feature points of the hard background region and the easy gap positive region of the feature image are mapped to a more distant space, thus constructing the first loss function.
[0045] In the specific implementation, after the server completes the region segmentation of the feature image, it selects infoNCE loss as the loss for contrastive learning. Based on infoNCE loss, feature points of hard positive regions and easy positive regions of the feature image are mapped to a similar space, feature points of easy gap positive regions and hard gap positive regions of the feature image are mapped to a similar space, and feature points of hard background regions and easy gap positive regions of the feature image are mapped to a more distant space, thus constructing the first loss function. Mapping to a similar space minimizes the infoNCE loss, while mapping to a more distant space maximizes the infoNCE loss. The existence of the first loss function strengthens the adversarial learning between positive and negative samples. The segmentation into five regions—hard background, easy gap positive, hard gap positive, hard positive, and easy positive—helps the model better distinguish between text and background regions, further improving the detection capability of the trained text detection model.
[0046] In one example, for any feature point (also called a sampling point) in any one of the five regions of the feature image, the feature vector of that feature point is used as the query vector q. For the sample image, sampling points of the five regions in the sample image are obtained in advance as sample points. This series of sample points can be represented as {k1,k2,…,k} after encoding. N}, where N is the total number of sample points, which are the keys of the dictionary. This dictionary contains only one unique k. + If we match q, then in contrastive learning, only q and k are considered. +During matching, the infoNCE loss value is minimized. The infoNCE loss function can be represented by the following function:
[0047]
[0048] In the formula, q is the query vector, representing the feature vector of a feature point in the feature map, and k + Let k be the vector of sample points in the sample image that match q. i Let N be the vector of the i-th sample point in the sample image, N be the total number of sample points, τ be the preset temperature coefficient, and L be the vector of the i-th sample point in the sample image. infoNCE This represents the loss value of the infoNCE loss function.
[0049] Step 103: Perform text segmentation on the sample image based on the text detection results of the sample image, determine the secondary labels corresponding to the segmented text regions, and construct a second loss function based on the secondary labels.
[0050] In the specific implementation, after obtaining the text detection results of the sample images output by the initial model, the server can perform text segmentation on the sample images based on the text detection results. The server determines the secondary labels corresponding to the segmented text regions and constructs a second loss function based on the secondary labels. The secondary labels determined by the server are either fuzzy labels or false positive labels. Fuzzy labels are used to characterize the fuzzy semantic clustering category of the segmented text regions, while false positive labels are used to characterize the segmented text regions as background interference information.
[0051] In one example, the server performs text segmentation on the sample image based on the text detection results, determines the secondary labels corresponding to the segmented text regions, and constructs a second loss function based on the secondary labels. This can be achieved through methods such as... Figure 4 The steps shown are implemented as follows:
[0052] Step 1031: Based on the text detection results of the sample image, perform pixel-level text segmentation on the sample image to obtain the segmented text regions and their corresponding text content.
[0053] In the specific implementation, the server can perform pixel-level text segmentation on the sample image based on the text detection results, resulting in a binary image where text pixels are labeled as foreground (i.e., the segmented text regions), and non-text pixels are labeled as background. The server then obtains the segmented text regions and their corresponding text content.
[0054] Step 1032: Based on the text content corresponding to the segmented text regions, perform fuzzy semantic clustering on the segmented text regions in a preset number of sample images, label the segmented text regions of the same fuzzy semantic clustering category with the corresponding fuzzy label, and label the segmented text regions whose corresponding text content is not real text with false positive labels.
[0055] In its implementation, after obtaining the segmented text regions and their corresponding text content, the server performs fuzzy semantic clustering on all text regions segmented from a preset number of sample images based on the text content of each segmented text region. It then labels the segmented text regions belonging to the same fuzzy semantic clustering category with fuzzy labels corresponding to that category, and labels the segmented text regions whose corresponding text content is not real text with false positive labels. For example, A to Z in an image can be clustered into one category, representing uppercase English letters, while a to z can be clustered into another category, representing lowercase English letters. Both uppercase and lowercase English letters are fuzzy labels, while the S-shaped pattern in the image is not real text, and the server labels it with a false positive label.
[0056] Step 1033: Map the fuzzy labels and false positive labels to a more distant space to construct the second loss function.
[0057] In its implementation, the model needs to distinguish between real text samples and false positive samples such as building patterns. Therefore, the server maps the fuzzy labels and false positive labels to a more distant spatial region, thus constructing the second loss function.
[0058] Step 104: Iteratively train the initial model until convergence using the first loss function and the second loss function to obtain the trained text detection model.
[0059] In the specific implementation, after the server obtains the first loss function and the second loss function, it can iteratively train the initial model based on the training sample set and according to the first loss function and the second loss function until convergence, thus obtaining the trained text detection model.
[0060] Step 105: Input the image to be detected into the text detection model to obtain the text detection result of the image to be detected.
[0061] In the specific implementation, after the server obtains the trained text detection model, it deploys the text detection model in the required use scenario. When the model is used, the image to be detected is input into the text detection model, and the text detection result of the image to be detected is obtained from the output of the text detection model.
[0062] In this embodiment, at the pixel level, compared to traditional text detection methods, it utilizes contrastive learning to divide the text region of the sample image into multiple pairs of different positive and negative samples. By learning the differences between these pairs, the discriminability of text features is improved. The trained text detection model can accurately extract text features from the image to be detected, achieving more precise detection of text boundaries. At the pixel block level, in challenging scenarios such as complex backgrounds and blurred text, it assigns secondary labels through fuzzy semantic clustering, accurately distinguishing real text samples from false positive samples such as building patterns. It possesses stronger ability to distinguish interference information, effectively improving the robustness and accuracy of text detection. The finally trained text detection model can achieve relatively accurate feature extraction for images containing text of arbitrary shapes, whether curved or tilted, significantly improving the universality and generalization ability of the text detection model. It has significant practical application value and can promote the development of the field of text image processing.
[0063] Another embodiment of this application proposes a text detection system that extracts features at the pixel block level based on contrastive learning. The implementation details of the text detection system that extracts features at the pixel block level based on contrastive learning proposed in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.
[0064] The specific structure of the text detection system based on contrastive learning for extracting features at the pixel level proposed in this embodiment can be as follows: Figure 5 As shown, the system includes a model training module 201 and a model usage module 202. The model training module 201 includes an acquisition unit 2011, a first loss function construction unit 2012, a second loss function construction unit 2013, and a training execution unit 2014.
[0065] The acquisition unit 2011 is used to acquire sample images, input the sample images into the initial model, obtain the feature images output by the backbone network of the initial model, and obtain the text detection results of the sample images output by the initial model based on the feature images.
[0066] The first loss function construction unit 2012 is used to divide the feature image into regions. Based on the idea of contrastive learning and the feature points of different regions of the feature image, the first loss function is constructed.
[0067] The second loss function construction unit 2013 is used to perform text segmentation on the sample image based on the text detection results of the sample image, determine the secondary label corresponding to the segmented text region, and construct the second loss function based on the secondary label. The secondary label is either a fuzzy label or a false positive label. The fuzzy label is used to characterize the fuzzy semantic clustering category of the segmented text region, and the false positive label is used to characterize the segmented text region as background interference information.
[0068] The training execution unit 2014 is used to iteratively train the initial model until convergence based on the first loss function and the second loss function, thereby obtaining the trained text detection model.
[0069] The model uses module 202 to input the image to be detected into the text detection model and obtain the text detection result of the image to be detected.
[0070] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and effects mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.
[0071] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.
[0072] In another embodiment, in order to verify the superiority of the text detection method based on contrastive learning for extracting features at the pixel block level proposed in this application, the server conducted simulation experiments on three models, including the text detection model trained in this application (hereinafter referred to as OURS).
[0073] 1. Simulation Experiment Conditions
[0074] This experiment is conducted using a central processing unit (CPU). This simulation experiment was conducted using MATLAB software on an i5-3470 3.2GHz CPU, 4GB of RAM, and a Windows 7 operating system. The data used in the simulation experiment came from publicly available datasets and self-collected datasets.
[0075] 2. Simulation Experiment Content
[0076] This experiment used a contrastive learning method to extract features at the pixel level and then verified whether the accuracy of text detection was improved. The simulation results are shown in Table 1. OURS's performance is significantly better than PixelLink and TextSnake.
[0077] Table 1: Performance Comparison of Simulation Results
[0078] Precision Recall F-measure PixelLink 83.0 73.2 77.8 TextSnake 83.2 73.9 78.3 OURS 87.3 78.8 82.8
[0079] Another embodiment of this application relates to an electronic device, such as... Figure 6 As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions executable by the at least one processor 301, the instructions being executed by the at least one processor 301 to enable the at least one processor 301 to perform the text detection method based on contrastive learning to extract features at the pixel block level in the above embodiments.
[0080] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.
[0081] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.
[0082] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.
[0083] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0084] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application.
Claims
1. A text detection method based on contrastive learning to extract features at the pixel-block level, characterized in that, include: The sample image is input into the initial model to obtain the feature image output by the backbone network of the initial model, and the text detection result of the sample image output by the initial model based on the feature image is obtained. The feature image is divided into regions, and a first loss function is constructed based on the contrastive learning approach and the feature points of different regions of the feature image. Based on the text detection results of the sample image, text segmentation is performed on the sample image to determine the secondary labels corresponding to the segmented text regions, and a second loss function is constructed based on the secondary labels; wherein, the secondary labels are fuzzy labels or false positive labels, the fuzzy labels are used to characterize the fuzzy semantic clustering category of the segmented text regions, and the false positive labels are used to characterize the segmented text regions as background interference information; The initial model is iteratively trained until convergence using the first loss function and the second loss function to obtain a trained text detection model; The image to be detected is input into the text detection model to obtain the text detection result of the image to be detected; The step of dividing the feature image into regions, and constructing a first loss function based on the contrastive learning approach and feature points of different regions of the feature image, includes: The text regions marked on the sample image are expanded outward once and contracted inward three times in sequence, resulting in hard background region, easy gap positive region, hard gap positive region, hard positive region and easy positive region from the outside to the inside. Based on the five regions on the sample image, the feature image is divided into regions; Based on infoNCE loss, feature points in the hard positive region and the easy positive region of the feature image are mapped to a similar space, feature points in the easy gap positive region and the hard gap positive region of the feature image are mapped to a similar space, and feature points in the hard background region and the easy gap positive region of the feature image are mapped to a more distant space, thus constructing a first loss function; wherein, mapping to a similar space minimizes the infoNCE loss, and mapping to a more distant space maximizes the infoNCE loss; The step of performing text segmentation on the sample image based on the text detection results of the sample image, determining the secondary labels corresponding to the segmented text regions, and constructing a second loss function based on the secondary labels includes: Based on the text detection results of the sample image, pixel-level text segmentation is performed on the sample image to obtain the segmented text regions and their corresponding text content. Based on the text content corresponding to the segmented text region, fuzzy semantic clustering is performed on the segmented text regions in a preset number of sample images. Fuzzy labels corresponding to the fuzzy semantic clustering category are labeled for the segmented text regions of the same fuzzy semantic clustering category, and false positive labels are labeled for the segmented text regions whose corresponding text content is not real text. The fuzzy labels and false positive labels are mapped to a more distant space to construct a second loss function.
2. The text detection method based on contrastive learning to extract features at the pixel block level according to claim 1, characterized in that, The offset for each outward expansion and inward contraction is determined by the following formula: ； in, The area of the annotated text region. The perimeter of the marked text region. The preset scaling factor. The offset is the specified amount.
3. The text detection method based on contrastive learning to extract features at the pixel block level according to claim 1, characterized in that, The infoNCE loss function is represented by the following function: ； in, The query vector represents the feature vector of a feature point in the feature map. For the sample image and the The vector of matched sample points, For the first in the sample image A vector of sample points The total number of the sample points. This is the preset temperature coefficient.
4. The text detection method based on contrastive learning for feature extraction at the pixel block level according to any one of claims 1 to 3, characterized in that, The sample image was obtained through the following steps: Acquire several images containing text and images without text as base source images; The base material image is preprocessed, including scaling, cropping and resolution normalization, and the preprocessed image is converted into a grayscale image. All the converted grayscale images are used as sample images to generate a training sample set.
5. The text detection method based on contrastive learning for feature extraction at the pixel block level according to any one of claims 1 to 3, characterized in that, The backbone network adopts a ResNet+FPN structure; The backbone network extracts features from the sample images through four branches, resulting in intermediate feature images of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 size. The intermediate feature images of 1 / 8, 1 / 16, and 1 / 32 sizes are upsampled respectively to obtain three intermediate feature images of 1 / 4 size; The four 1 / 4-sized intermediate feature images are stitched together in sequence to obtain the feature image output by the backbone network.
6. A text detection system for extracting features at the pixel-level based on contrastive learning, used to implement the text detection method for extracting features at the pixel-level based on contrastive learning as described in any one of claims 1 to 5, characterized in that, The system includes: a model training module and a model usage module. The model training module includes an acquisition unit, a first loss function construction unit, a second loss function construction unit, and a training execution unit. The acquisition unit is used to acquire sample images, input the sample images into the initial model, obtain the feature images output by the backbone network of the initial model, and obtain the text detection results of the sample images output by the initial model based on the feature images; The first loss function construction unit is used to divide the feature image into regions and construct a first loss function based on the contrastive learning idea and the feature points of different regions of the feature image; The second loss function construction unit is used to perform text segmentation on the sample image based on the text detection result of the sample image, determine the secondary label corresponding to the segmented text region, and construct a second loss function based on the secondary label. The secondary label is a fuzzy label or a false positive label. The fuzzy label is used to characterize the fuzzy semantic clustering category of the segmented text region, and the false positive label is used to characterize the segmented text region as background interference information. The training execution unit is used to iteratively train the initial model until convergence based on the first loss function and the second loss function to obtain a trained text detection model; The model uses modules to input the image to be detected into the text detection model to obtain the text detection result of the image to be detected.
7. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the text detection method based on contrastive learning to extract features at the pixel block level as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the text detection method based on contrastive learning to extract features at the pixel block level according to any one of claims 1 to 5.