An unsupervised scene text segmentation method based on undirected graph and key point detection
By using undirected graphs and keypoint detection, we have solved the problems of scene text segmentation's dependence on labeled data and small text processing, achieving fast and stable single-character segmentation and recognition while reducing computational resource consumption.
Patent Information
- Application Number
- CN202310354472.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-04
- Publication Date
- 2025-10-28
- Estimated Expiration
- 2043-04-04
AI Technical Summary
Existing scene text segmentation technologies require a large amount of labeled data, making it difficult to handle small texts and complex scenes. Furthermore, they are not effective at segmenting individual characters, resulting in high computational resource consumption.
We employ an undirected graph and keypoint detection method. By constructing a weighted undirected graph, we use the SIFT algorithm to detect keypoints, and then perform region segmentation centered on the keypoints. We extract geometric features to remove the background, retain the character regions, and combine the minimum spanning tree algorithm with post-processing steps for segmentation.
It reduces reliance on labeled data, improves processing speed and stability, is suitable for small datasets, can effectively segment and recognize individual characters, and reduces computational resource requirements.
Smart Images
Figure CN116403221B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to an unsupervised scene text segmentation method based on undirected graphs and key point detection. Background Technology
[0002] Scene text segmentation can extract text from an entire scene image, thereby helping computer vision tasks recognize text. Specifically, scene text segmentation can help with the following tasks:
[0003] (1) Text detection: Separating text and non-text regions makes text detection tasks more accurate in identifying text.
[0004] (2) Text recognition: Text segmentation can reduce noise and non-text interference, and improve the accuracy of text recognition.
[0005] (3) Text translation: For text scenarios that require translation, text segmentation can improve the performance of the translation system, thereby improving the accuracy and quality of the translation.
[0006] (4) Scene understanding: Segmenting a scene into text and non-text parts can help computers understand the scene and identify and infer the elements and relationships in the scene.
[0007] Therefore, scene text segmentation is of great significance for many computer vision and natural language processing tasks.
[0008] Currently, scene text segmentation technology is mainly based on deep learning methods, specifically including the following three directions:
[0009] (1) Convolutional Neural Network-based methods: Using convolutional neural networks to perform end-to-end training and segmentation of images. Examples include FCN (Fully Convolutional Network), UNet, and DeepLab.
[0010] (2) Attention-based approach: Introduce the attention mechanism into the neural network so that the network model can pay more attention to the text region.
[0011] (3) Semantic segmentation-based methods: For specific scenarios, such as roads and buildings, semantic information is used to assist in text segmentation.
[0012] Although existing scene text segmentation technologies have made great progress, they still have the following shortcomings:
[0013] (1) Deep learning-based scene text segmentation methods require a large amount of labeled data to train the model, and manual labeling is usually expensive.
[0014] (2) Current scene text segmentation methods mainly segment and detect a text region (such as an English word), without segmenting individual characters in the word. This increases the difficulty of scene text recognition and requires a more complex neural network model to achieve scene text recognition.
[0015] (3) Processing small text: Existing scene text segmentation technology still faces significant challenges for small fonts, low resolution, and blurry text. Summary of the Invention
[0016] The purpose of this invention is to at least solve one of the technical problems existing in the prior art, and to provide an unsupervised scene text segmentation method based on undirected graphs and key point detection, so as to solve the problems mentioned in the background art.
[0017] To achieve the above objectives, the present invention provides the following technical solution: an unsupervised scene text segmentation method based on undirected graphs and keypoint detection, S1; constructing an undirected graph of the entire image, the purpose of this step is to transform the image into an undirected graph, thereby facilitating subsequent segmentation processing,
[0018] S2; Use the SIFT algorithm encapsulated in the OpenCV library to perform key point detection on the image;
[0019] S3; Divide the region around each key point;
[0020] S4; Extract geometric features for each segmented region, remove the background region, and retain the character region. These features include aspect ratio and character edge width. Only regions whose aspect ratio and character edge width are less than a set threshold are identified as character regions.
[0021] S5; Post-processing section.
[0022] Preferably, the specific steps in S1 are:
[0023] Preprocessing: The original image is converted to grayscale and then Gaussian blurred to reduce image noise;
[0024] Node generation: Each pixel in the image is generated as a node, and these nodes are not connected by edges.
[0025] Edge generation: Calculate the similarity between each node and its surrounding pixels, and use the similarity as the weight of the edge between the two nodes. Similarity calculation methods include pixel intensity-based similarity calculation, color histogram-based similarity calculation, and gradient feature-based similarity calculation.
[0026] Weighted Undirected Graph Generation: Based on the node and edge information described above, a weighted undirected graph is constructed. In this undirected graph, nodes represent pixels in the image, edges represent the similarity between pixels, and the edge weights represent the similarity values between pixels.
[0027] Preferably, the specific step of S2 is: to create a SIFT object using the SIFT_create function of the xfeatures2d module of OpenCV;
[0028] Convert the input image to a grayscale image;
[0029] Use the detect function of the SIFT object to detect key points in the image.
[0030] Preferably, the specific steps of S3 are as follows:
[0031] Set two thresholds, MaxPixels and MaxSize. MaxPixels represents the maximum number of pixels that each text character region may contain, and MaxSize represents the maximum width and height of each text character region.
[0032] For the detected set of keypoints, the minimum spanning tree algorithm is applied to construct segmented regions, starting from each keypoint in the set. The specific steps are as follows:
[0033] Sort all edges within the MaxSize region (both width and height) in ascending order of weight;
[0034] Iterate through each edge in turn. If the two nodes connected by the edge are not in the same connected component, add the edge to the spanning tree; otherwise, ignore the edge. When adding an edge, the two nodes connected by the edge must be merged into the same connected component.
[0035] Repeat the steps until the number of edges in the spanning tree reaches MaxPixels-1, or until all remaining edges within the MaxSize region belong to another connected component. Ultimately, we obtain a minimum spanning tree containing the shortest paths connecting all nodes. Here, the length of the shortest path is the sum of the edge weights.
[0036] The minimum spanning tree corresponding to each key point is the segmented region.
[0037] Preferably, the post-processing portion described in S5 includes the following steps:
[0038] Remove small areas: This can be achieved by setting an area threshold.
[0039] Smooth boundaries: Smooth the boundaries of adjacent regions to eliminate noise and details on the boundaries;
[0040] Region merging: Merging adjacent regions into a larger region can be achieved by setting a similarity measurement method between regions;
[0041] Edge refinement: The segments are refined to make them clearer, which is beneficial for subsequent scene text recognition.
[0042] Compared with the prior art, the beneficial effects of the present invention are:
[0043] Compared to existing deep learning-based scene text segmentation methods, our method has the following main advantages:
[0044] (1) No need for a large amount of labeled data: Deep learning-based scene text detection methods require a large amount of labeled data to train the model, while undirected graph-based segmentation methods do not require a large amount of labeled data to detect scene text, thus greatly reducing the workload of manual annotation.
[0045] (2) Fast processing speed: The undirected graph-based segmentation method can quickly segment images, avoiding the problem that deep learning methods require a lot of computing resources (especially GPUs), and is suitable for running on devices with limited computing resources.
[0046] (3) Suitable for small datasets: For some scenarios where the text dataset is relatively small, the segmentation method based on undirected graphs can be more stable and reliable than the method based on deep learning. Attached Figure Description
[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments:
[0048] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation
[0049] This section will describe in detail specific embodiments of the present invention. Preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the drawings is to supplement the textual description with graphics, so that people can intuitively and vividly understand each technical feature and overall technical solution of the present invention, but they should not be construed as limiting the scope of protection of the present invention.
[0050] See also Figure 1 This invention provides a technical solution: an unsupervised scene text segmentation method based on undirected graphs and keypoint detection. 1. Constructing an undirected graph for the entire image. The purpose of this step is to transform the image into an undirected graph, thereby facilitating subsequent segmentation processing. The specific implementation steps are as follows:
[0051] 1.1 Preprocessing: The original image is converted into a grayscale image and Gaussian blurred to reduce image noise.
[0052] 1.2 Node generation: Each pixel in the image is generated as a node, and these nodes are not connected by edges.
[0053] 1.3 Edge Generation: Calculate the similarity between each node and its surrounding pixels, and use this similarity as the weight of the edge between the two nodes. Common similarity calculation methods include pixel intensity-based similarity calculation, color histogram-based similarity calculation, and gradient feature-based similarity calculation. Our invention uses the pixel intensity-based similarity calculation method.
[0054] 1.4 Weighted Undirected Graph Generation: Based on the node and edge information described above, a weighted undirected graph is constructed. In this undirected graph, nodes represent pixels in the image, edges represent the similarity between pixels, and the weight of the edges represents the similarity value between pixels.
[0055] 2. For keypoint detection in the image, we used the SIFT algorithm encapsulated in the OpenCV library. The specific steps are as follows:
[0056] 2.1 Use the SIFT_create function of OpenCV's xfeatures2d module to create a SIFT object;
[0057] 2.2 Convert the read-in image to a grayscale image;
[0058] 2.3 Use the detect function of the SIFT object to detect key points in the image.
[0059] 3. Divide the region into segments centered on each key point. The specific steps are as follows:
[0060] 3.1 Set two thresholds, MaxPixels and MaxSize. MaxPixels represents the maximum number of pixels that each text character region may contain, and MaxSize represents the maximum width and height of each text character region.
[0061] 3.2 For the set of key points detected in step 2, using each key point in the set as a starting point, the minimum spanning tree algorithm is applied to construct the segmentation region. The specific steps are as follows:
[0062] 3.2.1 Sort all edges within the MaxSize region (both width and height) in ascending order of weight.
[0063] 3.2.2 Iterate through each edge in turn. If the two nodes connected by the edge are not in the same connected component, add the edge to the spanning tree; otherwise, ignore the edge. When adding an edge, the two nodes connected by the edge need to be merged into the same connected component.
[0064] 3.2.3 Repeat step 2 until the number of edges in the spanning tree reaches MaxPixels-1, or all remaining edges within the MaxSize region belong to another connected component. Finally, we obtain a minimum spanning tree, which contains the shortest path connecting all nodes. Here, the length of the shortest path is the sum of the edge weights.
[0065] 3.2.4 The minimum spanning tree corresponding to each key point is the segmented region.
[0066] 4. Extract geometric features for each segmented region to remove background areas as much as possible and retain character regions. These features include aspect ratio and character edge width. Only regions with aspect ratio and character edge width less than a set threshold are identified as character regions.
[0067] 5. The post-processing section includes the following steps:
[0068] 5.1 Removing small areas: Removing small areas can be achieved by setting an area threshold.
[0069] 5.2 Smooth Boundaries: Smooth the boundaries of adjacent regions to eliminate noise and details on the boundaries.
[0070] 5.3 Region Merging: Merging adjacent regions into a larger region can be achieved by setting a similarity measurement method between regions.
[0071] 5.4 Edge Refinement: The segments are refined to make the edges clearer, which is beneficial for subsequent scene text recognition.
[0072] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.
Claims
1. An unsupervised scene text segmentation method based on undirected graphs and keypoint detection, characterized in that: S1; Construct an undirected graph of the entire image. The purpose of this step is to transform the image into an undirected graph, thereby facilitating subsequent segmentation processing. S2; Use the SIFT algorithm encapsulated in the OpenCV library to perform keypoint detection on the image; S3; Divide the region around each key point; S4; Extract geometric features for each segmented region, remove the background region, and retain the character region. These features include aspect ratio and character edge width. Only regions whose aspect ratio and character edge width are less than a set threshold are identified as character regions. S5; Post-processing section.
2. The unsupervised scene text segmentation method based on undirected graphs and keypoint detection according to claim 1, characterized in that: The specific steps in S1 are as follows: Preprocessing: The original image is converted to grayscale and Gaussian blurred to reduce image noise; Node generation: Each pixel in the image is generated as a node, and these nodes are not connected by edges. Edge generation: Calculate the similarity between each node and its surrounding pixels, and use the similarity as the weight of the edge between the two nodes. The similarity calculation methods include similarity calculation based on pixel intensity, similarity calculation based on color histogram, and similarity calculation based on gradient features. The invention uses the similarity calculation method based on pixel intensity. Weighted Undirected Graph Generation: Based on the node and edge information above, a weighted undirected graph is constructed. In this undirected graph, nodes represent pixels in the image, edges represent the similarity between pixels, and the weight of the edges represents the similarity value between pixels.
3. The unsupervised scene text segmentation method based on undirected graphs and keypoint detection according to claim 2, characterized in that: The specific steps of S2 are: using the SIFT_create function of the xfeatures2d module of OpenCV to create a SIFT object; Convert the read-in image to a grayscale image; Use the detect function of the SIFT object to detect key points in the image.
4. The unsupervised scene text segmentation method based on undirected graphs and keypoint detection according to claim 3, characterized in that: The specific steps of S3 are as follows: Set two thresholds, MaxPixels and MaxSize. MaxPixels represents the maximum number of pixels that each text character region may contain, and MaxSize represents the maximum width and height of each text character region. For the detected set of keypoints, the minimum spanning tree algorithm is applied to construct segmented regions, starting from each keypoint in the set. The specific steps are as follows: Sort all edges within the MaxSize region (both width and height) in ascending order of weight; Traverse each edge in turn. If the two nodes connected by the edge are not in the same connected component, add the edge to the spanning tree. Otherwise, ignore the edge. When adding an edge, the two nodes connected by the edge need to be merged into the same connected component. Repeat the steps until the number of edges in the spanning tree reaches MaxPixels-1 or all remaining edges in the MaxSize region belong to another connected component, eventually reaching a minimum spanning tree, which contains the shortest path connecting all nodes. Here, the length of the shortest path is the sum of the edge weights. The minimum spanning tree corresponding to each key point is the segmented region.
5. The unsupervised scene text segmentation method based on undirected graphs and keypoint detection according to claim 4, characterized in that: The post-processing section described in S5 includes the following steps: Remove small areas: This can be achieved by setting an area threshold. Smooth boundaries: Smooth the boundaries of adjacent regions to eliminate noise and details on the boundaries; Region merging: Merging adjacent regions into a larger region can be achieved by setting a similarity measurement method between regions; Edge refinement: The segments are refined to make them clearer, which is beneficial for subsequent scene text recognition.