A control net-based Chinese character stroke extraction method and device

Abstract

The application discloses a Chinese character stroke extraction method and device based on ControlNet, comprising the following steps: (1) extracting the stroke overlapping part of each Chinese character image from a plurality of font files and marking, pairing the Chinese character image and the corresponding marked image, constructing the training data set required for stroke overlapping area marking, and pre-training the ControlNet model; (2) inputting the original Chinese character image to be extracted into the trained ControlNet model, and predicting a stroke detection image of the marked stroke overlapping area; (3) according to the stroke overlapping area detection image, segmenting the stroke overlapping area and stroke segment from the original Chinese character image; (4) combining the stroke segments belonging to one stroke to obtain a complete independent stroke. By using the application, independent high-resolution strokes can be accurately extracted from high-resolution Chinese character images of various style fonts without using standard font data sets marked with stroke types.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing, and in particular relates to a method and apparatus for extracting Chinese character strokes based on ControlNet. Background Technology

[0002] Compared to most other writing systems, Chinese uses a vast character set, making Chinese font design extremely costly in terms of human resources. Although there are thousands upon thousands of character shapes in Chinese, the strokes, the smallest constituent units of a character, are only of 32 types, and each type of stroke is quite similar. This limitation in the shapes of Chinese strokes and the similarity among strokes of the same type make strokes play a crucial role in the structural analysis and writing instruction of Chinese characters. In scenarios where AI-assisted Chinese font design is used, designers need to extract individual stroke shapes from the whole character images generated by AI models to further optimize the design.

[0003] Stroke extraction refers to extracting each individual stroke from a complete Chinese character image. Automatic stroke extraction has always been a challenging problem in computer vision. This is primarily because the structural complexity of Chinese characters means that the number, type, and positional distribution of strokes vary across different characters. Secondly, although strokes of the same type share similarities, differences still exist between them. Finally, the diversity of Chinese font styles leads to significant variations in the strokes of the same character across different fonts. These factors make it extremely difficult to develop stroke extraction algorithms applicable to all Chinese characters and multiple font styles.

[0004] In recent years, with the continuous development of artificial intelligence technology, many studies and progress have been made in the extraction of Chinese character strokes based on deep learning methods such as semantic segmentation and instance segmentation.

[0005] For example, the method in "Liu L, Lin K, Huang S, et al. Instance Segmentation for Chinese Character Stroke Extraction, Datasets and Benchmarks[J]. arXiv preprintarXiv: 2210.13826, 2022." classifies strokes based on stroke annotation data of standard fonts and uses an instance segmentation algorithm for stroke segmentation. "Wang TQ, Jiang X, Liu C L. Query pixel guided stroke extraction with model-based matching for offline handwritten Chinese characters[J]. Pattern Recognition, 2022, 123: 108416." segments stroke fragments after detecting intersection points in the skeleton of handwritten strokes, and then uses a standard font stroke annotation dataset to perform stroke matching according to stroke category, but it can only handle handwritten Chinese characters with a resolution of 64x64. “Li M, Yu Y, Yang Y, et al. Strokeextraction of Chinese character based on deep structure deformable imageregistration[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(1): 1360-1367.” uses a generative adversarial network to perform style transfer on the already annotated strokes in the standard font, making them closer to the strokes of other font styles, and then performs image matching on the classification table.

[0006] The common characteristic of the above methods is that they all rely on a standard font stroke annotation dataset as a stroke template, extracting strokes from Chinese character images in other font types based on different stroke categories. These methods are only applicable to handwritten characters and low-resolution (no more than 256*256 pixels) Chinese character images. Constructing a standard font stroke annotation dataset requires not only individual stroke images for each Chinese character but also annotation of the corresponding stroke type. Therefore, these methods place high demands on the prior knowledge provided by the dataset. Their limitations include: if a stroke is rare or even missing in the stroke dataset, the stroke extraction effect will significantly decrease; furthermore, existing methods cannot handle high-resolution Chinese character images, and strokes extracted from low-resolution Chinese character images lose too much detail, making them unsuitable for intelligent font design scenarios.

[0007] In summary, there is currently a lack of general stroke extraction algorithms that do not rely on pre-built labeled stroke categories and can handle high-resolution font images. Summary of the Invention

[0008] This invention provides a method and apparatus for extracting Chinese character strokes based on ControlNet. It can extract independent high-resolution strokes from high-resolution Chinese character images of various styles of fonts with relatively high accuracy without using standard font datasets that label stroke types.

[0009] A method for extracting Chinese character strokes based on ControlNet includes the following steps: (1) Extract the overlapping parts of each Chinese character image from multiple font files and label them. Pair each Chinese character image with the corresponding labeled image to construct the training dataset required for the labeling of the overlapping parts of the strokes, and pre-train the ControlNet model. (2) Input the original Chinese character image to be extracted into the trained ControlNet model to predict the stroke detection image with the overlapping area of ​​the strokes marked; (3) Based on the stroke detection image with the stroke overlapping area marked, extract the stroke overlapping area and stroke fragments from the original Chinese character image; (4) Combine stroke segments that belong to one stroke to obtain a complete independent stroke.

[0010] This invention utilizes ControlNet in the image generation model to generate labeled images of overlapping stroke regions, thereby enabling the detection of overlapping stroke regions of Chinese characters. Then, stroke segments are segmented, and the stroke segments are recombined into complete and independent Chinese character strokes based on the slope direction of the Chinese character stroke skeleton at the overlapping region.

[0011] The specific process of step (1) is as follows: Extract the whole character vector outline of each Chinese character from multiple font files with stroke separation and rasterize it into a bitmap as the original Chinese character image, where the pixel value of the text part is 0 and the pixel value of the background part is 255; Meanwhile, the vector outline of each stroke of each Chinese character in the font file is obtained separately and rasterized into a bitmap to obtain an image with superimposed annotations of the overlapping areas of the strokes. The pixel value of the annotation image is set to 128 in the overlapping parts of the strokes, while the rest is consistent with the original Chinese character image mentioned above. Each Chinese character image is paired with its corresponding labeled image, with prompts consisting of a description of the corresponding font style or left blank, forming the training dataset required for labeling overlapping stroke regions, and the ControlNet model is pre-trained.

[0012] In step (1), the pre-trained ControlNet model is fine-tuned on a targeted dataset containing only the fonts whose strokes are to be extracted.

[0013] In step (2), the Chinese character image for which the strokes are to be extracted satisfies the following conditions: the background pixel value is 255 and the text pixel value is 0.

[0014] The specific process of step (3) is as follows: First, the original Chinese character images were analyzed. Calculate the distance from each pixel to the edge and exclude pixels with a distance less than 2 to generate a conditional mask. Set the conditional mask Stroke detection images of overlapping areas of strokes After performing element-wise matrix multiplication and binarization, the overlapping area map of strokes is obtained. ; Extracting the overlapping areas of strokes separately For each overlapping region, obtain a list of overlapping regions, Intersections; extract each overlapping region from the original Chinese character image. Eliminating the strokes yields an image of stroke fragments. Extract separately For each stroke segment in the list, check if the current stroke segment is connected to any overlapping area in the overlapping area list. If so, merge the overlapping area with the current stroke segment to obtain the original stroke segment list. .

[0015] The specific process of step (4) is as follows: (4-1) For each overlapping region in the Intersections list, check how many original stroke segments are connected to it. If there are two, it means that the overlapping region is generated by stroke connection, and the two original stroke segments are a valid combination. If there are more than two, it means that the overlapping region is generated by stroke intersection. In this case, combine the strokes connected to the overlapping region in pairs, and then use computer graphics to calculate the skeleton diagram and extract the skeleton at the overlapping region. ,in Represents the coordinates of skeletal points within the skeleton; (4-2) Check if there are any intersections in the skeleton. If there are, the combination of the two original strokes creates a corner, which does not belong to the same stroke and is an illegal combination. If there are no intersections, further use the difference method with a step size of 3 to calculate the list of the included angles corresponding to the slope of the tangent at each skeleton point in the overlapping area skeleton. ,Will Find the pairwise subtraction of all angles and calculate the maximum value. ;if A value greater than 40 degrees indicates that the skeleton at the overlapping area is an illegal combination; otherwise, it is considered a legal combination. By combining and judging all overlapping areas and their connected original stroke segments using the above method, a new list of combined stroke segments is obtained. ,in, Indicates the first An image of a combined stroke fragment. It is an index set representing that the combined stroke fragment is Depend on It is composed of the original stroke fragments corresponding to the index in the middle, if There is only one value inside. This indicates that the stroke segment has not been combined. (4-3) List of new combined stroke segments Each combination of stroke fragments and corresponding index list By continuously checking the index list of other stroke combinations in a loop. Does it meet the requirements? If so, it indicates a combination of stroke segments. and They are all composed of the same original stroke fragment, therefore and Further combinations are needed to ultimately obtain a complete list of individual strokes. .

[0016] In step (4-2), a list of the included angles corresponding to the slopes of the tangents at each skeleton point in the overlapping region skeleton is calculated using the difference method with a step size of 3. ,Will Find the pairwise subtraction of all angles and calculate the maximum value. The specific formula is as follows: In the formula, The total number of skeleton points. Represents the x and y coordinates of the i-th skeleton point. The angle between the tangents at the i-th skeleton point, calculated using the finite difference method. It is the arctangent function in the fourth quadrant, with a range of . .

[0017] A Chinese character stroke extraction device based on ControlNet includes a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they implement the above-mentioned Chinese character stroke extraction method.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention is characterized by high efficiency and accuracy. Compared with existing methods, it can extract independent high-resolution strokes from high-resolution Chinese character images of various styles of fonts with relatively high accuracy without using standard font datasets labeled with stroke types. For intelligent font design scenarios, the strokes extracted by this method can retain more image details, reducing the manual cost of font design optimization. Attached Figure Description

[0019] Figure 1 This is a flowchart of a Chinese character stroke extraction method based on ControlNet according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the input and output during ControlNet training and prediction in an embodiment of the present invention; Figure 3 This is a visualization of the steps involved in segmenting overlapping areas and stroke fragments from an original Chinese character image according to an embodiment of the present invention. Figure 4 This is a schematic diagram illustrating the combination of stroke overlap region type and stroke fragment at intersection point in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the combination of legal and illegal stroke segments in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the further combination of stroke segments that have the same original stroke fragments. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0021] like Figure 1 As shown, a method for extracting Chinese character strokes based on ControlNet mainly includes three parts: ControlNet stroke overlap region detection, segmentation of stroke overlap regions and stroke segments from the original Chinese character image, and piecing together stroke segments belonging to a single stroke into a complete stroke. The stroke overlap region refers to the overlapping area formed by the connection or intersection of different strokes in a Chinese character.

[0022] (a) ControlNet stroke overlap region detection For black and white Chinese character images, current common detection methods, such as semantic segmentation and instance segmentation, cannot obtain boundary information through color, resulting in unsatisfactory segmentation results for overlapping stroke regions. Furthermore, the need to downsample the input image leads to a loss of detail in the final result. Therefore, this technical solution uses a generative model instead of a segmentation model to detect overlapping stroke regions.

[0023] This invention uses the open-source Stable Diffusion and ControlNet as models for detecting overlapping stroke regions. ControlNet is a deep neural network structure that adds additional image conditions as input to the Stable Diffusion model, thereby controlling the model's output. Its training set consists of paired images and corresponding prompts. This invention constructs a custom annotation dataset for overlapping stroke regions, which does not require any additional annotation of stroke types during its construction.

[0024] In terms of dataset construction, this invention extracts the whole-character vector contours from the stroke-separated font files and rasterizes them into bitmaps as the original Chinese character images, where the pixel value of the text portion is 0 and the background portion is 255. Simultaneously, the vector contours of each stroke of each Chinese character in the font file are also separately obtained and rasterized into bitmaps, producing overlaid stroke annotation images—the pixel values ​​of the overlapping parts of the annotation images are set to 128, while the rest remain consistent with the aforementioned original Chinese character images. Pairing the original Chinese character images with the corresponding annotation images, with prompts either describing the corresponding font style or leaving them blank, constitutes the training dataset required for the annotation of overlapping stroke regions. This technique constructs a large-scale stroke overlapping region annotation dataset with over 100 fonts and over 280,000 paired data points. The pre-trained ControlNet model used later in this scheme has a training set bitmap size of 512×512 and a data scale of hundreds of thousands of data points.

[0025] When predicting overlapping areas of strokes, inputting a Chinese character image with a background pixel value of 255 and a text pixel value of 0 into the model allows for the inference of a stroke detection image that is similar in format to the labeled image. Figure 2 The input and output data of ControlNet during training and prediction are given.

[0026] Furthermore, to achieve better detection results on fonts of a specific style, the pre-trained model can be fine-tuned on a smaller, targeted dataset containing only that font. The fine-tuned model can better learn the features of overlapping stroke regions in the targeted dataset, thus achieving better results when detecting overlapping stroke regions of that specific font.

[0027] (ii) Segmenting overlapping areas and stroke fragments from the original Chinese character image. The purpose of stroke segmentation is to extract overlapping regions of strokes from the stroke overlap region detection image provided by ControlNet, and then segment the original stroke fragments from the original Chinese character image. To achieve this goal, this invention first processes the original Chinese character image... Calculate the distance from each pixel to the edge and exclude pixels with a distance less than 2 to generate a conditional mask. Set the conditional mask Image detection of overlapping areas with strokes Element-wise matrix multiplication is performed to suppress noise in the background and contour edges of the image output by ControlNet. Binarization yields a stroke overlap region map. ,in This represents element-wise multiplication of matrices. This represents the binarization operation.

[0028] For each overlapping region, a list of overlapping regions is obtained. ,in Indicates the first Images of overlapping regions. Each overlapping region is removed from the original Chinese character image to obtain images of stroke fragments. Similarly, extracting separately will For each stroke segment in the list, check if the current stroke segment is connected to any overlapping area in the overlapping area list. If so, merge the overlapping area with the current stroke segment to obtain the original stroke segment list. ,in Indicates the first An image of a stroke fragment. Figure 3 The visualization results of each step in stroke segmentation are presented.

[0029] (iii) Combine stroke fragments belonging to a single stroke into a complete stroke. There are generally two types of overlapping strokes. The first type is the stroke connection point formed by the connection of strokes. In this case, the stroke fragment after merging the overlapping area is a complete stroke and no further operation is required. The second type is the stroke intersection point formed by the intersection of strokes. In this case, the stroke fragment after merging the overlapping area still needs to be further combined to form a complete stroke. Figure 4 The differences between the two types of overlapping areas and the combination of stroke segments at the intersection points are given. To achieve stroke combination, this invention employs the following method: For For each overlapping region, check how many original stroke segments are connected to it. If there are fewer than two, no further processing is needed, as this indicates that the overlapping region was generated by stroke connections. If there are more than two, it indicates that the overlapping region was generated by stroke intersections. In this case, pairwise combine the strokes connected to the overlapping region, then use computer graphics to generate a skeleton diagram and extract the skeleton at the overlapping region. ,in This represents the coordinates of skeleton points within the skeleton. First, it checks if the skeleton has any intersections. If it does, the combination of two original strokes creates a corner, indicating they are not part of the same stroke and are therefore an illegal combination. If no intersections exist, it further uses a difference method with a step size of 3 to calculate the included angle corresponding to the slope of the tangent at each skeleton point in the overlapping area: Will Find the pairwise differences between the angles in the middle and calculate the maximum value. : if A value greater than 40 degrees indicates a significant angular change in the skeleton at the overlapping area, resulting in a corner, which is an illegal combination; conversely, a value less than 40 degrees indicates that the connection between the two original strokes is smooth and can be legally combined into a new stroke segment. Figure 5 Example diagrams of legal and illegal combinations of original stroke fragments and their overlapping areas are provided. By combining and judging all overlapping areas and their connected original stroke fragments using the above method, a new list of legally combined stroke fragments can be obtained. ,in Indicates the first The image of the combined stroke fragments It is an index set representing that the combined stroke fragment is Depend on The original stroke fragment corresponding to the index in the middle Combination, if There is only one value inside. This indicates that the stroke segment has not been combined, that is... .

[0030] Finally, for Each combination of stroke fragments and corresponding index list By continuously checking the index list of other stroke combinations in a loop. Does it meet the requirements? If yes, it indicates a combination of stroke segments. and They are all composed of the same original stroke fragment, therefore and Further combinations are also needed, such as Figure 6 As shown, this situation typically occurs when a stroke intersects with multiple strokes simultaneously, resulting in the overlapping area being divided into three or more original stroke segments. After combination, this produces more than one combined stroke segment. Ultimately, we can obtain... Where s'' w This is the final combination of complete individual strokes.

[0031] In the components of the custom dataset, the extraction of the overall vector and individual stroke vectors of Chinese characters from stroke-based fonts can be achieved using Python's fontTools library. For the image data input to ControlNet, the pixel values ​​are uniformly adjusted to between 0 and 1, and the pixel value type is converted to float32 for preprocessing. During the training of ControlNet, we specified a training worker count of 4, a batch size of 4, and a learning rate of 10. -5 During training, the loss value and training results for each epoch are saved to the log, and the 10 best-performing checkpoints are saved based on the loss value obtained during training.

[0032] During stroke segmentation, mask generation can utilize the `medial_axis` method from Python's `scikit-image` library to calculate the distance from each pixel in the original image to the edge. Extracting individual overlapping stroke regions and original stroke fragments can be done by using connected component labeling to extract each connected region in the image individually. Determining whether overlapping regions and original stroke fragments are connected can be achieved by using connected component labeling to determine if they form a single connected region.

[0033] During the stroke combination process, the skeleton of the combined stroke fragments can be extracted using a classic morphological skeletonization algorithm, specifically Zhang's skeletonization method. For the detection of skeleton intersections, a morphological hit-or-miss method can be used.

[0034] Based on the same inventive principle, this invention also provides a Chinese character stroke extraction device based on ControlNet, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the Chinese character stroke extraction method of the above embodiments.

[0035] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for extracting Chinese character strokes based on ControlNet, characterized in that, Includes the following steps: (1) Extract the overlapping parts of each Chinese character image from multiple font files and label them. Pair each Chinese character image with the corresponding labeled image to construct the training dataset required for the labeling of the overlapping parts of the strokes, and pre-train the ControlNet model. (2) Input the original Chinese character image to be extracted into the trained ControlNet model to predict the stroke detection image with the overlapping area of ​​the strokes marked; (3) Based on the stroke detection image with the stroke overlapping area marked, the stroke overlapping area and stroke fragments are segmented from the original Chinese character image; (4) Combine stroke segments that belong to one stroke to obtain a complete independent stroke.

2. The method for extracting Chinese character strokes based on ControlNet according to claim 1, characterized in that, The specific process of step (1) is as follows: Extract the whole character vector outline of each Chinese character from multiple font files with stroke separation and rasterize it into a bitmap as the original Chinese character image, where the pixel value of the text part is 0 and the pixel value of the background part is 255; Meanwhile, the vector outline of each stroke of each Chinese character in the font file is obtained separately and rasterized into a bitmap to obtain an image with superimposed annotations of the overlapping areas of the strokes. The pixel value of the annotation image is set to 128 in the overlapping parts of the strokes, while the rest is consistent with the original Chinese character image mentioned above. Each Chinese character image is paired with its corresponding labeled image, with prompts consisting of a description of the corresponding font style or left blank, forming the training dataset required for labeling overlapping stroke regions, and the ControlNet model is pre-trained.

3. The method for extracting Chinese character strokes based on ControlNet according to claim 1, characterized in that, In step (1), the pre-trained ControlNet model is fine-tuned on a targeted dataset containing only the fonts whose strokes are to be extracted.

4. The method for extracting Chinese character strokes based on ControlNet according to claim 1, characterized in that, In step (2), the Chinese character image for which the strokes are to be extracted satisfies the following conditions: the background pixel value is 255 and the text pixel value is 0.

5. The method for extracting Chinese character strokes based on ControlNet according to claim 1, characterized in that, The specific process of step (3) is as follows: First, the original Chinese character images were analyzed. Calculate the distance from each pixel to the edge and exclude pixels with a distance less than 2 to generate a conditional mask. Set the conditional mask Stroke detection images of overlapping areas of strokes After performing element-wise matrix multiplication and binarization, the overlapping area map of strokes is obtained. ; Extracting the overlapping areas of strokes separately For each overlapping region, obtain a list of overlapping regions, Intersections; extract each overlapping region from the original Chinese character image. Eliminating the strokes yields an image of stroke fragments. Extract separately For each stroke segment in the list, check if the current stroke segment is connected to any overlapping area in the overlapping area list. If so, merge the overlapping area with the current stroke segment to obtain the original stroke segment list. .

6. The method for extracting Chinese character strokes based on ControlNet according to claim 5, characterized in that, The specific process of step (4) is as follows: (4-1) For each overlapping region in the Intersections list, check how many original stroke segments are connected to it. If there are two, it means that the overlapping region is generated by stroke connection, and the two original stroke segments are a valid combination. If there are more than two, it means that the overlapping region is generated by stroke intersection. In this case, combine the strokes connected to the overlapping region in pairs, and then use computer graphics to calculate the skeleton diagram and extract the skeleton at the overlapping region. ,in Represents the coordinates of skeletal points within the skeleton; (4-2) Check if there are any intersections in the skeleton. If there are, the two original strokes combined to form a corner, which does not belong to the same stroke and is an illegal combination. If there are no intersections, then use the difference method with a step size of 3 to calculate the included angle corresponding to the slope of the tangent at each skeleton point in the overlapping area skeleton. ,Will Find the pairwise subtraction of all angles and calculate the maximum value. ; if A value greater than 40 degrees indicates that the skeleton at the overlapping area is an illegal combination; otherwise, it is considered a legal combination. By combining and judging all overlapping areas and their connected original stroke fragments using the above method, a new list of legally valid combined stroke fragments is obtained. ;in, Indicates the first An image of a combined stroke fragment. It is an index set representing that the combined stroke fragment is Depend on It is composed of the original stroke fragments corresponding to the index in the middle, if There is only one value inside. This indicates that the stroke segment has not been combined. (4-3) List of legally combined stroke segments Each combination of stroke fragments and corresponding index list By continuously checking the index list of other stroke combinations in a loop. Does it meet the requirements? If so, it indicates a combination of stroke segments. and They are all composed of the same original stroke fragment, therefore and Further combinations are needed to ultimately obtain a complete list of individual strokes. .

7. The method for extracting Chinese character strokes based on ControlNet according to claim 6, characterized in that, In step (4-2), a list of the included angles corresponding to the slopes of the tangents at each skeleton point in the overlapping region skeleton is calculated using the difference method with a step size of 3. ,Will Find the pairwise subtraction of all angles and calculate the maximum value. The specific formula is as follows: In the formula, The total number of skeleton points. Represents the x and y coordinates of the i-th skeleton point. The angle between the tangents at the i-th skeleton point, calculated using the finite difference method. It is the arctangent function in the fourth quadrant, with a range of . .

8. A Chinese character stroke extraction device based on ControlNet, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the Chinese character stroke extraction method according to any one of claims 1-7.

Images