A data construction method for stylized visual text image generation

Abstract

The application discloses a data construction method for style visual text image generation, comprising data acquisition; image preprocessing; inputting the visual text image into a multimodal large model to generate a text description; inputting the visual text image into a text recognition system and an image segmentation model to generate a text segmentation mask; inputting the text description and the visual text image into a visual semantic similarity model to verify semantic consistency; inputting the visual text image and the text segmentation mask into the multimodal large model to evaluate coverage accuracy; combining the text description, the text segmentation mask and the visual text image into a triple data; through constructing triple structure data containing a stylized text description, a visual text image and a pure text content image, the model effectively learns the corresponding relationship between the font style and the text content, and the data quality in the style visual text image generation task is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a data construction method for generating stylized visual text images. Background Technology

[0002] In recent years, visual text generation technology has been widely applied in practical scenarios such as poster design, advertising production, logo creation, artistic font generation, and digital media content production. Its core objective is to achieve controllable expression of font style while ensuring accurate text content generation. For this type of task, high-quality datasets are not only an important foundation for model training but also a key support for achieving joint modeling of text content and font style. However, existing visual text datasets still have significant shortcomings in terms of data composition, annotation methods, and style expression, making it difficult to meet the needs of complex stylized visual text generation tasks.

[0003] On the one hand, most existing datasets focus on text content recognition or OCR tasks, typically providing only text content, text detection boxes, character-level annotations, or simple image descriptions. Their design goals are more geared towards tasks like text detection, text recognition, or scene text understanding, rather than stylized visual text generation. Therefore, existing datasets generally lack explicit descriptions and structured annotations of visual attributes such as font color, texture, shape, typography, decorative elements, and background style, making it difficult for models to learn the correspondence between font style and text content. On the other hand, existing datasets often lack a complete "text content—font style—visual image" pairing structure, making it difficult to simultaneously provide the model with content information, style information, and text region spatial information. This makes it easy for the model to couple text semantics with font style during training, thus affecting the performance of the generated results in terms of style consistency, detail reproduction, and visual controllability.

[0004] In existing research on text image generation, the construction of high-quality datasets is a crucial foundation for improving model performance. For example, TextDiffuser's MARIO-10M dataset collects approximately 10 million images and employs an automated annotation process to perform tasks such as text detection, text recognition, and character-level segmentation, forming a relatively complete OCR annotation system. However, this dataset primarily relies on automated processes and templated description generation, resulting in text descriptions that suffer from limited expressive forms, insufficient semantic depth, and a lack of stylistic information, making it difficult to adapt to complex font styles and artistic text rendering scenarios. Furthermore, because it mainly focuses on text content recognition and character-level segmentation, it lacks explicit modeling of visual information such as font texture, color matching, decorative elements, and background style, thus failing to meet the demands of stylized visual text generation tasks for fine-grained style information.

[0005] The AnyWord-3M dataset is built upon NoahWukong, LAION-400M, and several other OCR recognition datasets, covering Chinese, English, and other multilingual texts. It utilizes an image captioning generation model to regenerate text descriptions from images, resulting in approximately 3.03 million images after filtering. While this dataset has advantages in terms of data scale, language coverage, and scene richness, it still heavily relies on the recognition and generation accuracy of the OCR system and image captioning generation model. Therefore, the generated text still has limitations in semantic coherence, detail accuracy, and natural layout. Furthermore, this type of dataset also lacks explicit annotations for font styles, decorative textures, and text region spatial structure, making it difficult to provide accurate style learning information for the model.

[0006] Most existing visual text datasets only provide a simple correspondence between images and text descriptions, lacking pixel-level segmentation information of text regions. This makes it difficult for models to accurately learn the spatial relationship between the main text region and the background region. When images contain complex backgrounds, decorative textures, multi-colored fonts, or artistic text effects, the model is easily interfered with by the background region, thus affecting the accuracy of font style modeling and text content generation. Therefore, relying solely on traditional OCR detection boxes or coarse-grained text region annotations is insufficient to meet the spatial constraints of text regions required for high-precision stylized visual text generation tasks.

[0007] In summary, there is an urgent need to construct a data construction method for stylized visual text image generation in visual text generation tasks, so as to form high-quality structured data containing information such as images, text descriptions and text segmentation masks, and provide reliable data support for visual text generation tasks with joint controllability of style and content. Summary of the Invention

[0008] (1) Technical problems to be solved This paper discloses a data construction method for stylized visual text image generation, which aims to solve the problem that existing visual text datasets are difficult to meet the requirements of complex stylized visual text generation tasks.

[0009] (2) Technical solution This invention provides a data construction method for generating stylized visual text images, comprising the following steps: Step 1: Collect visual text image data; Step 2: Preprocess the visual text image; Step 3: Input the preprocessed visual text image into the multimodal large model, and combine it with the preset prompt information to generate two levels of text description for each visual text image, including an overview-level title description and a detailed content description. Step 4: Input the preprocessed visual text image into the text recognition system and identify the text region. Then, use the image segmentation model to segment the identified text region and generate the corresponding pixel-level text segmentation mask. Step 5: Input the generated text description and the corresponding visual text image into the visual semantic similarity model to verify the semantic consistency between the text description and the visual text image; Step 6: Input the visual text image and the corresponding text segmentation mask into the multimodal large model to evaluate the coverage accuracy of the text segmentation mask and the actual text region in the visual text image. Step 7: Combine the text description that has passed semantic consistency verification, the text segmentation mask that has passed coverage accuracy evaluation, and the corresponding visual text image to form a complete triplet data structure. Steps 3 and 4 can be executed simultaneously or either can be executed first.

[0010] Furthermore, step 2 includes the following steps: Step 201: Perform file anomaly detection on the input visual text image to check if the image file is corrupted, unreadable, has an abnormal format, or has missing content, and remove abnormal image files; Step 202: Perform image size normalization processing on the visual text images that pass the file anomaly detection, and adjust the images of different sizes to a preset size; Step 203: Use an aesthetic evaluation model to score the image quality of the size-normalized visual text images, and remove image files with scores below a preset threshold.

[0011] Furthermore, step 3 includes the following steps: Step 301: Input the preprocessed visual text image into the multimodal large model InternVL2.5-38B-MPO; Step 302: Based on preset prompts, guide the multimodal large model to jointly understand the text content, font style, color, layout, decorative elements and background features in the visual text image; Step 303: Generate a summary-level heading description that briefly summarizes the main visual text content and overall style of the visual text image; Step 304: Generate a detailed description of the font style, text content, color scheme, layout, decorative elements, and background features in the visual text image.

[0012] Furthermore, step 4 includes the following steps: Step 401: Input the preprocessed visual text image into the text recognition system PaddleOCR; Step 402: Detect all text regions in each visual text image using PaddleOCR; Step 403: Identify and record the coordinate information of each text region to obtain the text region location information; Step 404: Input the text region location information into the image segmentation model TeXRNet; Step 405: Based on the text region location information, perform pixel-level segmentation of the text region in the visual text image; Step 406: Generate a text segmentation mask corresponding to the text strokes or text regions in the visual text image.

[0013] Furthermore, the segmentation process in step 4 is as follows: First, the image segmentation model uses a feature extraction backbone network to extract feature maps of visual text images, and then performs convolution mapping and normalization on the feature maps to obtain an initial text region prediction map. Next, a key feature pooling mechanism is introduced to extract category-level key feature matrices based on high-confidence text regions in the initial text region prediction map; Finally, an attention similarity check mechanism is used to calculate the text region attention map by combining the feature map and the key feature matrix. Then, the text region attention map, feature map and visual text image are fused, convolved and refined and converted into a text segmentation mask.

[0014] Furthermore, the key feature pooling process is as follows: Based on the initial text region prediction map, cosine similarity is calculated for the prediction responses of different categories; Based on the similarity results, the initial text region prediction map is adjusted by category confidence and dynamic bias is introduced to obtain the corrected semantic response map; The semantic response map is normalized to obtain a weight map for key feature pooling, and then the feature map is weighted and pooled to obtain the corresponding key feature vector. The key feature vectors of each category are concatenated to obtain the key feature matrix.

[0015] Furthermore, the attention similarity checking mechanism and the text segmentation mask generation process are as follows: Based on the attention similarity check mechanism, the feature map is used as the query and the key feature matrix is ​​used as the key. The text region attention map is calculated by dot product normalization. The text region attention map, feature map, and visual text image are fused and refined through several convolutional layers to obtain the final text segmentation result. The final text segmentation result is then used to obtain a text segmentation mask.

[0016] Furthermore, step 5 includes the following steps: Step 501: Input the generated text description and the corresponding visual text image into the visual semantic similarity model CLIP-ViT-L / 14; Step 502: Extract the visual feature vector of the visual text image and the semantic feature vector of the text description respectively using the visual semantic similarity model; Step 503: Calculate the similarity between the visual feature vector and the semantic feature vector; The similarity is compared with a preset threshold. When the similarity is equal to or higher than the preset threshold, the text description is determined to be consistent with the visual text image content; when the similarity is lower than the preset threshold, the text description is determined to be inconsistent with the visual text image content. Step 504: For text descriptions that are determined to be inconsistent, the text descriptions are then manually modified.

[0017] Furthermore, step 6 includes the following steps: Step 601: Input each visual text image and its corresponding text segmentation mask into the multimodal large model Gemini2.5; Step 602: Analyze the pixel-level coverage between the text segmentation mask and the text region in the image using Gemini 2.5; Step 603: Evaluate whether the text segmentation mask completely covers the text region in the image; Step 604: Calculate the matching degree between the text segmentation mask boundary and the actual text contour in the image, and compare the matching degree with the preset boundary threshold; Step 605: Determine whether the text segmentation mask incorrectly covers non-text background areas in the image; Step 606: When the matching degree is lower than the preset boundary threshold, the text segmentation mask does not completely cover the text area, or the text segmentation mask mistakenly covers the non-text background area, the text segmentation mask is determined to be unqualified. Step 607: Mark the text segmentation masks that are determined to be unqualified, and regenerate the corresponding text segmentation masks; Step 608: Repeat steps 601 to 607 until the text segmentation mask is deemed acceptable, or the preset maximum number of iterations is reached; Step 609: If the problem of inconsistency between the visual text image and the text segmentation mask is still not resolved after reaching the preset maximum number of iterations, the text segmentation mask will be manually corrected.

[0018] Furthermore, step 7 includes the following steps: Step 701: Save the text description that has passed the semantic consistency verification as a file in a specified format, including JSON format; Step 702: Obtain the visual text image corresponding to the text description; Step 703: Obtain the text segmentation mask corresponding to the visual text image; Step 704: Construct the visual text image, text description, and text segmentation mask into a triplet data structure.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: By generating hierarchical text descriptions through a multimodal large model and combining text recognition, text region segmentation, and visual semantic consistency verification, the model achieves joint annotation of font style, text content, and spatial region information in visual text images. Simultaneously, by constructing a ternary structure dataset containing stylized text descriptions, visual text images, and plain text content images, the model can effectively learn the correspondence between font style and text content. This addresses the problems of existing visual text datasets lacking explicit style representation, insufficient image-text alignment accuracy, and inaccurate text region annotation, thereby improving data quality and model generation performance in stylized visual text image generation tasks. Attached Figure Description

[0020] Figure 1 This is a flowchart of the present invention; Figure 2 This is a flowchart of the TexRNet backbone network in step 4 of the present invention.

[0021] Figure 3 This is the backbone network diagram of CLIP-ViT-L / 14 in step 5 of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] refer to Figure 1 This invention proposes a data construction method for generating stylized visual text images, specifically including the following steps: Step 1: Collect visual text images to be processed and construct the original dataset to provide basic materials for data construction; Step 2: Preprocess the input visual text image to obtain a high-quality visual text image. The specific steps are as follows: Step 201: Perform file anomaly detection on the input visual text image to determine if the image file is corrupted, cannot be read normally, has an abnormal file format, has missing image content, or incomplete image channels; for image files that are detected as abnormal, remove them directly from the original dataset to avoid abnormal samples affecting the subsequent model processing flow; Step 202: Perform image size normalization processing on the visual text images that pass the document anomaly detection, and uniformly adjust the images with different resolutions and aspect ratios to a preset size or a preset size range to meet the input requirements of subsequent multimodal large models, text recognition systems and image segmentation models. Step 203: Use an aesthetic evaluation model to score the image quality of the size-normalized visual text images, and remove image files with scores below a preset threshold from the original dataset to obtain high-quality visual text images that meet the visual quality requirements.

[0024] Step 3: Input the preprocessed visual text images into the multimodal large model, and combine them with preset prompts to generate two levels of text descriptions for each visual text image: an overview-level title description and a detailed content description. This transforms the original visual text images into text descriptions with hierarchical semantic information. The specific steps are as follows: Step 301: Input the preprocessed visual text image into the multimodal large model InternVL2.5-38B-MPO; Step 302: Based on preset prompts, guide the multimodal large model to jointly understand the text content, font style, color, layout, decorative elements and background features in the visual text image; Step 303: Generate a summary-level heading description that briefly summarizes the main visual text content and overall style of the visual text image; Step 304: Generate a detailed description of the font style, text content, color scheme, layout, decorative elements, and background features in the visual text image.

[0025] Step 4: Input the preprocessed visual text image into the text recognition system and identify the text region. Then, use the image segmentation model to segment the identified text region and generate the corresponding pixel-level text segmentation mask. Step 401: Input the preprocessed visual text image into the text recognition system PaddleOCR; Step 402: Detect all text regions in each visual text image using PaddleOCR; Step 403: Identify and record the coordinate information of each text region to obtain the text region location information; Step 404: Input the text region location information into the image segmentation model TeXRNet; Step 405: Based on the text region location information, perform pixel-level segmentation of the text region in the visual text image; Step 406: Generate a text segmentation mask corresponding to the text strokes or text regions in the visual text image.

[0026] It should be noted that, since visual text images usually contain complex backgrounds, decorative patterns, shadow textures and diverse font styles, relying solely on text recognition systems can only obtain the rough position or bounding box coordinates of the text area, making it difficult to accurately depict the strokes of the text, the outlines of the characters and the boundaries of non-rectangular text. Therefore, it is necessary to further introduce an image segmentation model to refine the text region at the pixel level. That is, the text region location information obtained by PaddleOCR is input into the image segmentation model TexRNet. The text region location information is used as a spatial prior to constrain the model to focus on candidate text regions in the image, thereby reducing the interference of complex backgrounds, decorative patterns and non-text regions on the segmentation results.

[0027] For further explanation of the segmentation process in step 4, please refer to [link / reference]. Figure 2 The image segmentation model TexRNet segments the preprocessed visual text images. As input, the feature extraction backbone network extracts features from the input image to obtain feature maps. :

[0028] in, This represents the feature extraction backbone network. This represents a deep semantic feature map extracted from a visual text image; Subsequently, convolutional layers are used to map the feature maps to initial text segmentation semantic maps. :

[0029] in, Indicates the convolution mapping operation. Indicates the bias term. Represents the initial text segmentation semantic graph; Furthermore, through The normalization function yields the initial text region prediction map:

[0030] This initial text region prediction map is used to distinguish text regions from background regions in a visual text image and serves as the basis for subsequent text feature refinement.

[0031] Specifically, because visual text images typically exhibit diverse characteristics in terms of color, texture, scale, and font style, low-confidence text regions are prone to exist in the initial text region prediction image. To address this, the image segmentation model TeXRNet introduces a key feature pooling mechanism to extract category-level key feature matrices from high-confidence text regions in the initial text region prediction image. The specific steps are as follows: Based on the initial text region prediction map ,in Indicates the number of categories. This represents the number of pixels in a single channel. Cosine similarity is calculated for the predicted responses of different categories.

[0032] in, This represents the formula for calculating the cosine similarity matrix. Represents the cosine similarity matrix. Represents the initial text region prediction map The Middle Predicted response graph of the class, Indicates the first Class and the The cosine similarity of the predicted responses between classes; the higher the similarity, the easier it is to confuse the prediction results of the two classes. Based on the similarity results, the model first uses the cosine similarity matrix... Prediction map of the initial text region The category confidence is adjusted by introducing a dynamic bias (New Bias) to suppress easily confused regions between categories, thus obtaining the corrected semantic response map. ;

[0033] in, Indicates dynamic bias operation; Subsequently, through Function response graph to the modified semantic response Normalization is performed to obtain a weight map for key feature pooling. Next, the feature map Perform weighted pooling to obtain the first Key feature vectors of a class :

[0034] in, Represents the first key feature pooling method. Class weight graph; Then, the key feature vectors of each category are concatenated to obtain the key feature matrix. :

[0035] in, Indicating the number of categories, this process can extract representative font texture and stroke structure information from high-confidence text regions to guide the recovery of low-confidence text regions; Furthermore, the image segmentation model TexRNet employs an attention similarity check mechanism to integrate feature maps. As a query, the key feature matrix As the key, the text region attention map is calculated using dot product normalization:

[0036] in, This represents a text region attention map. Indicates the number of categories. This indicates the number of pixels in a single channel; the attention map is used to reactivate text pixel regions that were ignored or had low confidence in the initial prediction stage, enabling the model to further identify potential text regions based on similar textures, similar stroke shapes, or similar character structures. Subsequently, the text region attention map Feature map and visual text images The data is then fused and refined through several convolutional layers to obtain the final text segmentation result. :

[0037] in, This indicates a convolutional thinning segmentation network. Indicates a splicing operation; Finally, this final text segmentation result is used. It is further converted into a text segmentation mask, which is used to identify the pixel positions in the image that belong to text strokes, character outlines or main text areas; It should be noted that there is no strict restriction on the order of steps 3 and 4 in this embodiment; they can be executed simultaneously or either one can be executed first.

[0038] Step 5: Input the text description generated in Step 3 and the corresponding visual text image into the visual semantic similarity model to verify the semantic consistency between the text description and the visual text image; the specific steps are as follows: Step 501: Input the text description generated in step 3 and the corresponding visual text image into the visual semantic similarity model CLIP-ViT-L / 14; Step 502: Extract the visual feature vector of the visual text image and the semantic feature vector of the text description respectively using the visual semantic similarity model; Step 503: Calculate the similarity between the visual feature vector and the semantic feature vector to obtain the semantic matching degree between the text description and the corresponding visual text image; The similarity is compared with a preset threshold. When the similarity is equal to or higher than the preset threshold, the text description is determined to be consistent with the visual text image content; when the similarity is lower than the preset threshold, the text description is determined to be inconsistent with the visual text image content. Step 504: For text descriptions that are determined to be inconsistent, the text descriptions are then manually modified.

[0039] Specifically, step 503 includes the following steps: Step 5031: Calculate the similarity between the visual feature vector and the semantic feature vector; Step 5032: Set the preset threshold to 0.7; Step 5033: When the similarity is equal to or higher than 0.7, the text description is determined to be consistent with the visual text image content; when the similarity is lower than 0.7, the text description is determined to be inconsistent with the visual text image content.

[0040] For details, please refer to the processing procedure in step 5. Figure 3 First, the visual text image and its corresponding text description are input into the visual semantic similarity model CLIP-ViT-L / 14, which includes an image encoder (CLIP Image Encoder) and a text encoder (CLIP Text Encoder). Then, the visual feature vector of the preprocessed visual text image is extracted by the image encoder. At the same time, the text description is segmented and encoded using a text tokenizer and input into the text encoder to extract the semantic feature vector of the text. Specifically, for the input visual text image Visual feature vectors are obtained through an image encoder:

[0041] in, This indicates the CLIP image encoder. Represents visual feature vectors. Indicates the feature embedding dimension; For the input text description After word segmentation and encoding, a semantic feature vector is obtained through a text encoder:

[0042] in, This indicates the CLIP text encoder. This represents a semantic feature vector.

[0043] Furthermore, to eliminate the impact of differences in feature vector scales on similarity calculation, the visual feature vectors and semantic feature vectors were respectively... Normalization process yields the normalized feature representation:

[0044] in, express Normalization operation, This represents the normalized visual feature vector. This represents the normalized semantic feature vector; Subsequently, the normalized visual feature vectors and semantic feature vectors are mapped to a unified semantic embedding space, and the semantic matching degree between the normalized visual feature vectors and semantic feature vectors is calculated using cosine similarity.

[0045] Because the normalized eigenvectors satisfy:

[0046] Therefore, cosine similarity can be further expressed as:

[0047] in, Representing visual text images With text description Semantic similarity score between them; Furthermore, the semantic similarity score is compared with a preset threshold of 0.7. When the text description matches the visual text image content, it indicates that the text description can well reflect the main text content and visual style features in the visual text image; when If so, it is determined that the text description is inconsistent with the visual text image content; Through the above process, CLIP-ViT-L / 14 can model the cross-modal correspondence between visual text images and text descriptions using a unified semantic embedding space, thereby achieving automated semantic consistency verification of text descriptions generated by multimodal large models and improving the alignment accuracy between images and text descriptions.

[0048] Step 6: Input each visual text image and its corresponding text segmentation mask into the multimodal large model Gemini2.5 to jointly understand the visual text image and the text segmentation mask. By analyzing the pixel-level correspondence between the mask region and the text region in the image, the coverage accuracy of the text segmentation mask and the actual text region in the image is evaluated. The specific steps are as follows: Step 601: Input each visual text image and its corresponding text segmentation mask into the multimodal large model Gemini2.5; Step 602: Analyze the pixel-level coverage between the text segmentation mask and the text region in the image using Gemini 2.5; Step 603: Evaluate whether the text segmentation mask completely covers the text region in the image; Step 604: Calculate the matching degree between the text segmentation mask boundary and the actual text contour in the image, and compare the matching degree with the preset boundary threshold; Step 605: Determine whether the text segmentation mask incorrectly covers non-text background areas in the image; Step 606: When the matching degree is lower than the preset boundary threshold, the text segmentation mask does not completely cover the text area, or the text segmentation mask mistakenly covers the non-text background area, the text segmentation mask is determined to be unqualified. Step 607: Mark the text segmentation masks that are determined to be unqualified, and regenerate the corresponding text segmentation masks; Step 608: Repeat steps 601 to 607 until the text segmentation mask is deemed acceptable, or the preset maximum number of iterations is reached; Step 609: If the problem of inconsistency between the visual text image and the text segmentation mask is still not resolved after reaching the preset maximum number of iterations, the text segmentation mask will be manually corrected.

[0049] Specifically, when determining whether the text segmentation mask is qualified, the model needs to determine whether the text segmentation mask completely covers the main text area and whether there are problems such as missing character strokes, partial omission of characters, or area breakage. The model also needs to analyze the degree of matching between the text segmentation mask boundary and the actual text outline, and compare the degree of matching with a preset boundary threshold to determine whether the text segmentation mask boundary is consistent with the text edge of the text region. In addition, the model also needs to determine whether the text segmentation mask mistakenly covers non-text background areas, such as decorative patterns, shadows, background textures, or non-text objects; When the text segmentation mask boundary matching degree is lower than the preset boundary threshold, does not completely cover the text area, or mistakenly covers the non-text background area, the text segmentation mask is deemed unqualified, and it is marked and the corresponding text segmentation mask is regenerated. Repeat the mask evaluation and regeneration process until the text segmentation mask is deemed acceptable or the preset maximum number of iterations is reached; if the mask and text region are still inconsistent after the maximum number of iterations is reached, the text segmentation mask will be manually corrected. Step 6 can further improve the boundary accuracy, coverage integrity, and region accuracy of the text segmentation mask, thereby ensuring that the final constructed data sample has reliable pixel-level text region annotation.

[0050] Step 7: Save the text description that has passed semantic consistency verification in a specified format, and combine the corresponding visual text image and the text segmentation mask that has passed the coverage accuracy evaluation to form a complete triple data structure. The triple data structure covers various font types such as standard fonts, artistic fonts, and handwritten fonts, and contains rich information on color, texture, typography, and background style. The specific steps are as follows: Step 701: Save the text description that has passed the semantic consistency verification as a file in a specified format, including JSON format; Step 702: Obtain the visual text image corresponding to the text description; Step 703: Obtain the text segmentation mask corresponding to the visual text image; Step 704: Construct the visual text image, text description, and text segmentation mask into a triplet data structure.

[0051] Specifically, each triplet data sample consists of three parts: a stylized text description with text content and background details, a source visual text image, and a text segmentation mask of a plain text content image. The stylized text description is used to represent text content, font style, color, decorative elements, and background information, while the source visual text image is used to provide a realistic visual text scene. The text segmentation mask is used to preserve the main text content and weaken background interference.

[0052] The triplet dataset constructed using this invention can provide a complete "style-content-image" ternary correspondence for stylized visual text image generation tasks, thereby improving the model's ability to jointly model font style and text content, and enhancing the performance of the generated results in terms of style consistency, text accuracy, and visual realism.

[0053] The technical principle of the method of this invention is: to generate hierarchical text descriptions through multimodal large models, and to achieve joint annotation of font style, text content and spatial region information in visual text images by combining text recognition, text region segmentation and visual semantic consistency verification. Meanwhile, by constructing a ternary structure data that includes stylized text descriptions, visual text images, and plain text content images, the model can effectively learn the correspondence between font styles and text content. This solves the problems of existing visual text datasets lacking explicit style representations, insufficient image-text alignment accuracy, and inaccurate text region annotation, thereby improving data quality and model generation performance in stylized visual text image generation tasks.

[0054] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0055] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] The technical features of the above embodiments can be combined arbitrarily. Furthermore, the numbering of each step is not intended to constrain the order of the steps; their order is permissible as long as there are no strict constraints on the sequence. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, any combination of these technical features that does not contradict each other should be considered within the scope of this specification.

[0057] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data construction method for generating stylized visual text images, characterized in that: Includes the following steps: Step 1: Collect visual text image data; Step 2: Preprocess the visual text image; Step 3: Input the preprocessed visual text image into the multimodal large model, and combine it with the preset prompt information to generate two levels of text description for each visual text image, including an overview-level title description and a detailed content description. Step 4: Input the preprocessed visual text image into the text recognition system and identify the text region. Then, use the image segmentation model to segment the identified text region and generate the corresponding pixel-level text segmentation mask. Step 5: Input the generated text description and the corresponding visual text image into the visual semantic similarity model to verify the semantic consistency between the text description and the visual text image; Step 6: Input the visual text image and the corresponding text segmentation mask into the multimodal large model to evaluate the coverage accuracy of the text segmentation mask and the actual text region in the visual text image. Step 7: Combine the text description that has passed semantic consistency verification, the text segmentation mask that has passed coverage accuracy evaluation, and the corresponding visual text image to form a complete triplet data structure. Steps 3 and 4 can be executed simultaneously or either can be executed first. Step 4 includes the following steps: Step 401: Input the preprocessed visual text image into the text recognition system PaddleOCR; Step 402: Detect all text regions in each visual text image using PaddleOCR; Step 403: Identify and record the coordinate information of each text region to obtain the text region location information; Step 404: Input the text region location information into the image segmentation model TeXRNet; Step 405: Based on the text region location information, perform pixel-level segmentation of the text region in the visual text image; Step 406: Generate a text segmentation mask corresponding to the text strokes or text regions in the visual text image; Step 6 includes the following steps: Step 601: Input each visual text image and its corresponding text segmentation mask into the multimodal large model Gemini2.5; Step 602: Analyze the pixel-level coverage between the text segmentation mask and the text region in the image using Gemini 2.5; Step 603: Evaluate whether the text segmentation mask completely covers the text region in the image; Step 604: Calculate the matching degree between the text segmentation mask boundary and the actual text contour in the image, and compare the matching degree with the preset boundary threshold; Step 605: Determine whether the text segmentation mask incorrectly covers non-text background areas in the image; Step 606: When the matching degree is lower than the preset boundary threshold, the text segmentation mask does not completely cover the text area, or the text segmentation mask mistakenly covers the non-text background area, the text segmentation mask is determined to be unqualified. Step 607: Mark the text segmentation masks that are determined to be unqualified, and regenerate the corresponding text segmentation masks; Step 608: Repeat steps 601 to 607 until the text segmentation mask is deemed acceptable, or the preset maximum number of iterations is reached; Step 609: If the problem of inconsistency between the visual text image and the text segmentation mask is still not resolved after reaching the preset maximum number of iterations, the text segmentation mask will be manually corrected.

2. The data construction method for stylized visual text image generation according to claim 1, characterized in that, Step 2 includes the following steps: Step 201: Perform file anomaly detection on the input visual text image to check if the image file is corrupted, unreadable, has an abnormal format, or has missing content, and remove abnormal image files; Step 202: Perform image size normalization processing on the visual text images that pass the file anomaly detection, and adjust the images of different sizes to a preset size; Step 203: Use an aesthetic evaluation model to score the image quality of the size-normalized visual text images, and remove image files with scores below a preset threshold.

3. The data construction method for generating stylized visual text images according to claim 1, characterized in that, Step 3 includes the following steps: Step 301: Input the preprocessed visual text image into the multimodal large model InternVL2.5-38B-MPO; Step 302: Based on preset prompts, guide the multimodal large model to jointly understand the text content, font style, color, layout, decorative elements and background features in the visual text image; Step 303: Generate a summary-level heading description that briefly summarizes the main visual text content and overall style of the visual text image; Step 304: Generate a detailed description of the font style, text content, color scheme, layout, decorative elements, and background features in the visual text image.

4. The data construction method for generating stylized visual text images according to claim 1, characterized in that, The segmentation process in step 4 is as follows: First, the image segmentation model uses a feature extraction backbone network to extract feature maps of visual text images, and then performs convolution mapping and normalization on the feature maps to obtain an initial text region prediction map. Next, a key feature pooling mechanism is introduced to extract category-level key feature matrices based on high-confidence text regions in the initial text region prediction map; Finally, an attention similarity check mechanism is used to calculate the text region attention map by combining the feature map and the key feature matrix. Then, the text region attention map, feature map and visual text image are fused, convolved and refined and converted into a text segmentation mask.

5. The data construction method for generating stylized visual text images according to claim 4, characterized in that, The key feature pooling process is as follows: Based on the initial text region prediction map, cosine similarity is calculated for the prediction responses of different categories; Based on the similarity results, the initial text region prediction map is adjusted by category confidence and dynamic bias is introduced to obtain the corrected semantic response map; The semantic response map is normalized to obtain a weight map for key feature pooling, and then the feature map is weighted and pooled to obtain the corresponding key feature vector. The key feature vectors of each category are concatenated to obtain the key feature matrix.

6. The data construction method for stylized visual text image generation according to claim 5, characterized in that, The attention similarity checking mechanism and the text segmentation mask generation process are as follows: Based on the attention similarity check mechanism, the feature map is used as the query and the key feature matrix is ​​used as the key. The text region attention map is calculated by dot product normalization. The text region attention map, feature map, and visual text image are fused and refined through several convolutional layers to obtain the final text segmentation result. The final text segmentation result is then used to obtain a text segmentation mask.

7. The data construction method for generating stylized visual text images according to claim 1, characterized in that, Step 5 includes the following steps: Step 501: Input the generated text description and the corresponding visual text image into the visual semantic similarity model CLIP-ViT-L / 14; Step 502: Extract the visual feature vector of the visual text image and the semantic feature vector of the text description respectively using the visual semantic similarity model; Step 503: Calculate the similarity between the visual feature vector and the semantic feature vector; The similarity is compared with a preset threshold. When the similarity is equal to or higher than the preset threshold, the text description is determined to be consistent with the visual text image content; when the similarity is lower than the preset threshold, the text description is determined to be inconsistent with the visual text image content. Step 504: For text descriptions that are determined to be inconsistent, the text descriptions are then manually modified.

8. The data construction method for generating stylized visual text images according to claim 1, characterized in that, Step 7 includes the following steps: Step 701: Save the text description that has passed the semantic consistency verification as a file in a specified format, including JSON format; Step 702: Obtain the visual text image corresponding to the text description; Step 703: Obtain the text segmentation mask corresponding to the visual text image; Step 704: Construct the visual text image, text description, and text segmentation mask into a triplet data structure.

Images