A multi-modal guided controllable creative font generation method and system

Abstract

This disclosure provides a multimodal guided method and system for generating controllable creative fonts, relating to the field of artistic font generation technology. The method includes: acquiring target text, target font, and multimodal data representing user intent; generating conversion prompts and artistic design prompts based on the multimodal data; extracting multiple paths from the target text and target font, selecting the path with the highest similarity according to the conversion prompts for glyph transformation, and obtaining the transformed image; utilizing a multi-mask guided diffusion process, and using artistic design prompts as prompts, generating the final artistic font image for the transformed image. This invention, through multimodal guidance and creative texture generation using a diffusion model, overcomes the limitations of existing methods in artistic font generation, providing better solutions, particularly in artistic expression and readability, precise control, and multilingual adaptability.

Description

Technical Field

[0001] This disclosure relates to the field of artistic font generation technology, specifically to a multimodal guided controllable creative font generation method and system. Background Technology

[0002] Artistic typeface generation transforms characters into visually expressive forms, aiming to enhance artistry or convey specific meanings. It combines ordinary characters with stylistic and decorative elements to create font designs that are both legible and attractive. This fusion of visual and semantic elements effectively captures the viewer's attention and strengthens message delivery. In recent years, artistic typeface generation has been widely applied in advertising design, typography, and brand identity, achieving remarkable results.

[0003] With the rise of multimodal large language models, researchers are attempting to combine the strengths of large language models and large vision models to solve more complex multimodal tasks. Large language models excel in language understanding and reasoning, while large vision models perform exceptionally well in visual tasks (such as image segmentation), especially when guided by language cues, effectively generating images that meet visual requirements. Multimodal information refers to inputs that simultaneously contain multiple different types of data, such as text, images, audio, and video. In this context, multimodal artistic font generation integrates these different information sources, performs glyph transformation and style transfer, and creates artistic, creative, and expressive text graphics.

[0004] However, despite the existing technologies' ability to achieve glyph conversion and style transfer, several challenges remain:

[0005] (1) The balance between artistic expression and readability

[0006] Existing methods are mostly constrained by fixed frameworks and concepts, lacking sufficient flexibility and adaptability, especially when dealing with cultural backgrounds and linguistic diversity, making it difficult to achieve a good balance between artistic expression and readability. This results in visually unappealing outputs that sacrifice readability.

[0007] (2) Lack of precise control, including background and area textures

[0008] Current end-to-end generation methods typically generate characters directly without offering sufficient flexibility for designers to participate in personalized adjustments. This limits the fine-grained management of background control and region textures in artistic glyph generation. Furthermore, many methods (such as WordArtDesigner and MetaDesigner) introduce complex backgrounds and artifacts during the generation process, leading to increased complexity in fonts and applications.

[0009] (3) Language restrictions

[0010] Current methods for generating artistic fonts mainly focus on Latin letters. When dealing with non-Latin scripts (such as Chinese, Japanese, and Korean), many methods still face significant difficulties, such as TextDiffuser, TextDiffuser2, AnyText, and FontDiffuser. This limits the application of these methods in multilingual and multicultural environments, failing to meet user needs.

[0011] Therefore, existing artistic font generation technologies are insufficient in terms of artistic expression, readability, precise control, and cross-language adaptability. Summary of the Invention

[0012] To address the aforementioned issues, this disclosure proposes a multimodal guided method and system for generating controllable creative fonts. By using multimodal guidance and creative texture generation through a diffusion model, it overcomes the limitations of existing methods in artistic font generation, providing better solutions, particularly in terms of artistic expression and readability, precise control, and multilingual adaptability.

[0013] According to some embodiments, the present disclosure adopts the following technical solutions:

[0014] A multimodal guided method for generating controllable creative fonts includes:

[0015] Acquire the target text, target font, and multimodal data representing user intent;

[0016] Based on multimodal data, generate transformation prompts and art design prompts;

[0017] Multiple paths are extracted from the target text and target font. Based on the conversion prompts, the path with the highest similarity is selected for glyph transformation to obtain the transformed image.

[0018] By utilizing a multi-mask-guided diffusion process and employing artistic design cues as hints, the final artistic glyph image is generated for the transformed image.

[0019] According to some embodiments, the present disclosure adopts the following technical solutions:

[0020] A multimodal guided controllable creative font generation system, comprising:

[0021] The data acquisition module is configured to acquire target text, target font, and multimodal data used to represent user intent.

[0022] The prompt generation module is configured to generate transformation prompts and art design prompts based on multimodal data;

[0023] The glyph transformation module is configured to: extract multiple paths from the target text and target font, select the path with the highest similarity according to the transformation prompts, perform glyph transformation, and obtain the transformed image;

[0024] The image generation module is configured to generate the final artistic letterform image for the transformed image by using a multi-mask-guided diffusion process and artistic design cues as prompts.

[0025] According to some embodiments, the present disclosure adopts the following technical solutions:

[0026] A computer program product includes a computer program that, when executed by a processor, implements the aforementioned multimodal guided controllable creative font generation method.

[0027] According to some embodiments, the present disclosure adopts the following technical solutions:

[0028] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the aforementioned multimodal guided controllable creative font generation method.

[0029] According to some embodiments, the present disclosure adopts the following technical solutions:

[0030] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the multimodal guided controllable creative font generation method.

[0031] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0032] This invention provides a multimodal guided, multi-level controlled artistic typography framework. Combined with an iterative feedback mechanism, it addresses the key challenge of balancing creativity, readability, and adaptability, while also supporting non-Latin alphabet typography systems. Notably, this framework is training-independent and user-driven. While enhancing artistic expression, it simplifies the generation process into a series of sequential steps by incorporating design principles and generative models. The framework transforms the input through iterative feedback, continuously optimizing the generated results and ensuring an organic integration of creativity and readability.

[0033] This invention employs a Chain Thinking (CoT) method based on Multimodal Large Language Model (MLLM), which extracts design features from multimodal inputs and integrates user intent to transform abstract concepts into concrete and actionable prompts, thereby improving the effect of artistic expression.

[0034] To achieve multi-level control, this invention extracts vector paths aligned with multimodal inputs under multi-level control to ensure output consistency. Furthermore, it implements a texture diffusion process that maintains background stability, enabling precise control over variations in shape, color, and texture while ensuring readability. Simultaneously, the diffusion process addresses language limitations.

[0035] To provide better control and consistent output, this invention designs a feedback module that includes a scoring mechanism, label-based feedback, and an automated feedback mechanism. This module supports iterative optimization to ensure that the generated output meets high-quality aesthetic standards. Attached Figure Description

[0036] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0037] Figure 1 This is a schematic diagram of the model structure of Example 1. Detailed Implementation

[0038] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0039] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0040] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0041] Example 1

[0042] One embodiment of this disclosure provides a multimodal guided method for generating controllable creative fonts, including:

[0043] Step S1: Obtain the target text, target font, and multimodal data used to represent user intent;

[0044] Step S2: Generate transformation prompts and art design prompts based on multimodal data;

[0045] Step S3: Extract multiple paths from the target text and target font, select the path with the highest similarity according to the conversion prompts, perform glyph transformation, and obtain the transformed image;

[0046] Step S4: Using a multi-mask-guided diffusion process, artistic design cues are used as prompts to generate the final artistic letterform image for the transformed image.

[0047] As one embodiment, this disclosure discloses a multimodal guided controllable creative font generation method. Utilizing a generative model, it controls the generation of creative fonts from input target text, target font, and multimodal data representing user intent, ultimately yielding artistic font images that meet user expectations. Through multimodal guidance and the use of a diffusion model for creative texture generation, it overcomes the limitations of existing methods in artistic font generation, particularly providing better solutions in artistic expression and readability, precise control, and multilingual adaptability. The generative model here, such as... Figure 1 As shown, it includes four modules: multimodal intent extraction module, automatic path matching and conversion module, background-preserving texture generation module, and iterative prompt feedback optimization module. Taking the target text "Guilin" and the target font "fontname.ttf" as an example, the four modules are explained.

[0048] I. Multimodal Intent Extraction Module

[0049] Multimodal data refers to user-provided data that represents user intent. Text, images, audio, and video all revolve around the target text and font. For example, the text and audio might describe Guilin: "Karst peaks of various shapes stand in the distance, their slopes covered in lush vegetation. The sky is ablaze with colorful clouds, a blend of blue-purple and pink-orange hues, creating a dreamlike atmosphere. The calm water, like a mirror, perfectly reflects the sky and mountains, creating a serene and ethereal scene. Houses can be seen along the shore, adding a touch of life to this natural landscape. Overall, it evokes a sense of tranquility, remoteness, and breathtaking beauty."

[0050] This module extracts user intent from multimodal data, transforming abstract concepts into concrete generation prompts. From the input multimodal data (such as text, images, audio, and video), the system can analyze and generate specific artistic font design prompts. For concrete concepts in multimodal data (such as "peach" or "maple tree"), the system can effectively generate matching artistic fonts. However, for abstract concepts in multimodal data (such as "Baroque" or "cold"), detailed descriptions are often difficult to provide, requiring innovative prompt analysis and generation strategies to obtain more nuanced descriptions.

[0051] To analyze and extract specific prompts, a prompt analysis and generation strategy based on a large language model was designed. The chain reasoning (CoT) method of the large language model is used to extract and generate specific prompts through task-specific templates (i.e., the prompt engineering of the large language model).

[0052] Specifically, it receives multimodal input from the user, which is used to represent the user's intent. Specifically, it first receives the user's multimodal input U. p The input elements ∈{text, image, audio, video} are fed into a multimodal intent extraction template containing multiple design dimensions. Utilizing a large language model and cue engineering, conversion cue P related to the user's intent is generated progressively. trans Art and Design Tips P art .

[0053] For conversion prompt P trans By recognizing keywords in the input (such as "mountain"), these keywords are used as initial transformation prompts. Subsequent iterative optimizations can adjust the transformation prompts to precisely control the glyph transformation and ensure alignment with the expected output.

[0054] For P art A multi-step sub-hint generation chain was designed, with each sub-hint corresponding to a specific design dimension (e.g., "semantic theme", "font style", "layout and structure", "color and tone", "shape and structure") to comprehensively guide the generation process of artistic fonts. By decomposing the generation process, it ensures that all relevant design elements are covered, and supports users to make real-time fine-tuning through interaction, so as to adjust each design dimension, such as color, shape or layout, according to the user's personalized needs.

[0055] II. Automatic Path Matching and Transformation Module

[0056] This module is primarily responsible for extracting paths from the input target text and target font, and then converting them according to the conversion prompts P. trans Automatically select the most suitable path for glyph conversion, specifically:

[0057] 1. Automatic level path matching

[0058] Analysis of conversion prompt P trans The process of selecting the most suitable path from multiple levels (such as strokes, paths, characters, and words) is divided into two stages:

[0059] (1) Path preprocessing

[0060] First, using the open-source font engine library FreeType, glyph outlines are extracted from the user-input text and TrueType font, and each outline is converted into a cubic Bézier curve. Then, the Bézier curves are standardized using vectorization commands. Finally, the path of each character is segmented to generate the original path set W, ensuring that non-Latin characters maintain shape consistency during the conversion process.

[0061] Specifically, based on the user's target text U w and target font U fn Obtain the image I of the target text. ori .

[0062] Decompose the path into the original path set W:

[0063] First, use the outline extraction function from the open-source font engine library FreeType. Extract the outline of each character; for consistent processing, use a function. The outline is converted into a cubic Bézier curve. A Bézier curve is a commonly used mathematical representation to depict the outline of a font. There are three main types of curves in the outline of a font: linear, quadratic Bézier curve, and cubic Bézier curve. For convenience and uniformity, both linear and quadratic Bézier curves are converted into cubic Bézier curves.

[0064] Then, through the function Convert the Bézier chain to a vector format, i.e., the path command in SVG format. <path>These commands use a combination of instructions such as M (move to point) and C (cubic Bézier curve) with control points to describe the character outline, then add the necessary content to form an SVG file. If the input is a string, multiple characters are converted into SVG files, and each character generates a separate SVG file. Then, the font is scaled and aligned, and the typography of the entire string is normalized to ensure that the characters are correctly centered within the given canvas size.

[0065] Next, the text and characters are segmented from a vector perspective (Path Dec.). Each path is used as the smallest segmentation unit, and the path is segmented using the "M" command to form the basic segmentation unit L. i For non-Latin characters, the number of control points that make up the character (string) is pre-calculated to ensure smoother transformations and better shape rendering consistency;

[0066] Finally, the original path set W is obtained:

[0067]

[0068] (2) Automatic route selection

[0069] Use CLIP encoder to convert the prompt P trans Each path is compared in a high-dimensional feature space, and their similarity is calculated. The path with the highest similarity is selected for conversion. This process ensures that the selected path visually matches the expected conversion prompt while preserving the readability of the glyphs.

[0070] Specifically, firstly, a differentiable grating, DiffVG, is used. The original path set W is rasterized. Then, the transformation cue P is processed using the encoder ε of the contrastive language-image pre-trained model CLIP. trans and each rasterized path L in the original path set W i Encode them, map them to the same high-dimensional feature space, and compute all composable ones. and P trans The cosine similarity score between the paths is used. Simultaneously, to ensure sufficient deformation capability, the number of commands for each path is pre-calculated to assess its complexity, and paths with small areas or low confidence are filtered out. If the number of commands for a path is greater than or equal to a certain threshold, the path is considered to have high complexity, can exhibit richer shape variations, and better meet user expectations. Finally, the set of paths with the highest similarity scores above this threshold is selected for transformation; this local transformation method maintains the readability of the output.

[0071] The optimal path set L is selected as follows:

[0072]

[0073] Among them, F m (·) represents the automatic level path matching function, and cos represents the calculation of the cosine similarity score.

[0074] 2. Character transformation

[0075] Convert the optimal path set L to the transformation hint P. trans Aligned deformation path set

[0076] First, the optimal path set L is used to generate the corresponding raster image I. Then, image I is optimized and cropped to generate a randomly enhanced image. Next, the conversion prompt P will be displayed. trans and The input is fed into a frozen visual-language model, and the optimal path set L is applied to the transformation cue P by calculating the shape diffusion similarity loss (SDS Loss). trans Alignment as a deformable path set under guidance Deformation path set The transformation is performed through the following process:

[0077]

[0078] Among them, F t (·) represents the character shape transformation function.

[0079] Deformation path set Align with the original path set W (Path Ali.) and rasterize. Combine them (Path Com.) to generate the deformed image I mod To preserve the character structure, L and... The center points and size differences between them are adjusted using bounding boxes; finally, the deformed image I... mod The following is generated:

[0080]

[0081] III. Texture and Background Preservation Module

[0082] To ensure background stability and precise texture control during the artistic font generation process, this module uses the U-Net model, involving two stages: noise addition and denoising. The noise addition stage employs a multi-mask-guided diffusion process, in which the transformed image is used. As input, multiple masks are used to add noise, ensuring precise control of the region and maintaining the stability of the background; in the denoising stage, noise is gradually removed during the back diffusion process, ultimately generating artistic fonts that meet the user's expectations. During this noise reduction process, the feedback module repeatedly optimizes the art design prompts P. art Specifically:

[0083] 1. Add noise

[0084] Image I mod Convert to the latent space representation z0 and use the region mask M reg To constrain the noise region, the region mask M reg This facilitates artistic expression, ensures background stability and texture accuracy, and allows for control over specific areas, thereby precisely controlling the positioning of artistic lettering.

[0085] Image I mod Transformed into a latent space representation via a variational autoencoder (VAE) ε. Then, use the region mask. This is used to constrain noisy areas, thereby precisely controlling the position of artistic typography.

[0086] Two methods were designed to extract the mask:

[0087] (1) Automatic path alignment: from the aligned path set Contour information is extracted and downsampled to adjust to the dimensions of the latent space.

[0088] (2) User Interaction: The Segment Anything Model is used to extract precise masks and filter out masks that are too large or too small. An interactive mode is also provided, allowing users to generate an M by clicking and selecting. reg This enhances the accuracy and personalization of the process.

[0089] By element-wise multiplication, the region mask M is... reg This is applied to the latent space representation z0; during the forward process, Gaussian noise of time step t is added to the latent representation z0 to obtain the initial noisy latent representation x. in :

[0090]

[0091] To maintain background stability during the generation process, an inverse mask is defined. M reg ′ =1-M reg Then, M reg ′ Element-wise multiplication with z0 yields a noise-free latent representation x.

[0092] At each time step t∈{1,...,T}, Gaussian noise corresponding to the time step t is added to the latent representation x, thus obtaining the noise latent representation x in the background. bg :x bg =q(x t |x).

[0093] Next, x in With x bg By splicing the data, we obtain the latent noise representation x at time step t. t .

[0094] 2. Noise Reduction

[0095] Introducing a global mask Ensure clear edges in the background, specifically:

[0096] Using art and design tips P art As a hint, and by introducing a global mask As conditional information c, to preserve sharp edges of the background, the predicted latent representation is obtained through the following denoising process.

[0097]

[0098] Where, ∈ θ (·,·) represents the noise predicted by the U-Net network.

[0099] This denoising process is repeated at each time step t∈{ε,...,1} until the predicted latent representation x0 is obtained, and then passed through the decoder. Convert it into output image I output It should be noted that x bg Obtained through a forward process, while x in Optimization is achieved through a noise reduction process.

[0100] IV. Iterative Feedback Optimization Module

[0101] To further optimize the output results, this embodiment introduces a feedback module that combines user feedback and automated feedback. Based on user feedback and tag feedback, this module adjusts the template (i.e., the prompt of the large language model) in the multimodal intent extraction module and works collaboratively through a weighted adjustment mechanism to gradually refine and continuously optimize the generated results.

[0102] Specifically, this includes user feedback and tag feedback. User feedback involves users rating and tagging elements to guide the system in adjusting related design elements, such as color, shape, and texture. For example, if a user rates color low, the weight of color-related prompts will be increased. The automatic feedback section uses tools such as the CLIP model to automatically score the output and adjust generation prompts, thereby continuously improving the quality of the model output. User feedback and automatic feedback interact to ensure that the generation process not only adapts to the user's personalized needs but also improves the overall design quality. Through this mechanism, precise control of the artistic font generation process is achieved, enabling more customized and high-quality output based on user requirements.

[0103] User feedback is further divided into rating feedback and tag feedback. The following is an explanation of the three types of feedback:

[0104] (1) Rating feedback: Users rate the various aspects of the generated data, prompting the model to adjust relevant elements based on the feedback. For example, if the score for "color" is low, the model will pay more attention to color in subsequent generation processes and prioritize the low-scoring parts through a weighted system, thereby optimizing the output effect.

[0105] (2) Tag Feedback: Users can tag dimensions they are dissatisfied with (e.g., material, shape, etc.) to guide the model to focus on these aspects. This targeted feedback helps the model adjust content according to the user's specific preferences, improving personalization and accuracy.

[0106] (3) Automated feedback: The output results are evaluated using automatic scoring tools (such as CLIP, EasyOCR, etc.) and the generated prompts are automatically adjusted. Automated feedback complements user feedback input and can make more extensive quality adjustments to the generated results, thereby ensuring continuous optimization of the output.

[0107] These feedback mechanisms are interconnected. User feedback provides specific guidance, while automated feedback ensures overall quality adjustments, thereby achieving more accurate and personalized output results.

[0108] To verify the performance of the method in this embodiment, it was compared with DALLE 3 and Stable Diffusion in FontDiffuser, AnyText, TextDiffuser, TextDiffuser-2, and ChatGPT-4o. The generated images were quantitatively evaluated using four metrics: CLIP score, DINO score, OCR confidence, and Hausdorff Distence (HD). CLIP score and DINO score were used to evaluate style consistency and similarity in the generated output, while OCR confidence and Hausdorff Distence (HD) were used to evaluate contour similarity. Table 1 shows the comparison results of these metrics.

[0109] Table 1

[0110]

[0111] As can be seen, AnyText and TextDiffuser performed poorly across the various evaluation metrics, particularly in CLIP score and Hausdorff distance (HD), failing to effectively balance style and readability. This is mainly due to the limitations of their training data, making it difficult for them to generate diverse and accurate styles, especially when dealing with non-Latin characters. In contrast, DALLE 3 and Stable Diffusion performed well in CLIP and DINO scores, capturing style consistency relatively well, but scored lower in OCR confidence and HD metrics, indicating that while they performed well in style alignment, they had issues with the accuracy of glyph structure, easily leading to text shape distortion.

[0112] The method in this embodiment outperforms other models in all metrics, demonstrating excellent performance. It ensures consistency in style, richness in semantics, and accuracy in glyph structure, thereby generating aesthetically pleasing and highly readable artistic typography that provides more precise and diverse artistic font effects.

[0113] Example 2

[0114] One embodiment of this disclosure provides a multimodal guided controllable creative font generation system, comprising:

[0115] The data acquisition module is configured to acquire target text, target font, and multimodal data used to represent user intent.

[0116] The prompt generation module is configured to generate transformation prompts and art design prompts based on multimodal data;

[0117] The glyph transformation module is configured to: extract multiple paths from the target text and target font, select the path with the highest similarity according to the transformation prompts, perform glyph transformation, and obtain the transformed image;

[0118] The image generation module is configured to generate the final artistic letterform image for the transformed image by using a multi-mask-guided diffusion process and artistic design cues as prompts.

[0119] Example 3

[0120] One embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned multimodal guided controllable creative font generation method.

[0121] Example 4

[0122] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the multimodal guided controllable creative font generation method.

[0123] Example 5

[0124] One embodiment of this disclosure provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the multimodal guided controllable creative font generation method.

[0125] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0127] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.< / path>

Claims

1. A method for generating controllable creative fonts under multimodal guidance, characterized in that, include: Acquire the target text, target font, and multimodal data representing user intent; Based on multimodal data, generate transformation prompts and art design prompts; Multiple paths are extracted from the target text and target font. Based on the conversion prompts, the path with the highest similarity is selected for glyph transformation to obtain the transformed image. By utilizing a multi-mask-guided diffusion process and employing artistic design cues as hints, the final artistic glyph image is generated for the transformed image.

2. The method for generating controllable creative fonts with multimodal guidance as described in claim 1, characterized in that, The conversion prompt is generated by identifying keywords in multimodal data and generating conversion prompts; The art design prompts are generated by using a sub-prompt generation chain to generate sub-prompts under different design dimensions.

3. The method for generating controllable creative fonts with multimodal guidance as described in claim 1, characterized in that, The extraction of multiple paths from the target text and target font specifically involves: Extract glyph outlines from the target text and target font, and convert each outline into a cubic Bézier curve; The Bézier curve is standardized using the vectorization command; The path of each character is segmented to generate the original path set.

4. The method for generating controllable creative fonts with multimodal guidance as described in claim 3, characterized in that, The step of selecting the path with the highest similarity based on the transformation prompt involves using the CLIP encoder to compare the transformation prompt with each path in the original path set in a high-dimensional feature space, calculating the similarity between them, and selecting the path with the highest similarity to form the optimal path set.

5. The method for generating controllable creative fonts with multimodal guidance as described in claim 4, characterized in that, The glyph transformation involves converting the optimal path set into modified paths aligned with the transformation prompts, aligning the modified paths with the original path set, and combining them through rasterization to generate the transformed image.

6. The method for generating controllable creative fonts with multimodal guidance as described in claim 1, characterized in that, The multi-mask-guided diffusion process is specifically as follows: The transformed image is converted into a latent spatial representation, and a region mask is used to constrain the noisy region. Introduce a global mask to ensure clear edges on the background; By progressively denoising through a reverse process, an artistic font image that meets the user's expectations is finally generated.

7. A multimodal guided controllable creative font generation system, characterized in that, include: The data acquisition module is configured to acquire target text, target font, and multimodal data used to represent user intent. The prompt generation module is configured to generate transformation prompts and art design prompts based on multimodal data; The glyph transformation module is configured to: extract multiple paths from the target text and target font, select the path with the highest similarity according to the transformation prompts, perform glyph transformation, and obtain the transformed image; The image generation module is configured to generate the final artistic letterform image for the transformed image by using a multi-mask-guided diffusion process and artistic design cues as prompts.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal guided controllable creative font generation method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement a multimodal guided controllable creative font generation method as described in any one of claims 1-6.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform a multimodal guided controllable creative font generation method as described in any one of claims 1-6.

Images