Hair body generation method and device based on diffusion model

Abstract

The invention discloses a hair body generation method and device based on a diffusion model. The method comprises the steps of obtaining a coding vector and semantic information of each Chinese character in a character library; training a diffusion model by taking the calligraphy image set of the general grass body as a training sample and combining the coding vector of each Chinese character as a priori condition to obtain a first-stage training result of the diffusion model; on the basis of the first-stage training result, a hair-body-based calligraphy image set serves as a training sample, semantic information of each Chinese character serves as a priori condition, a diffusion model is trained, and a second-stage training result of the diffusion model is obtained; and generating a hair body generation model based on the second-stage training result. According to the method, a cursive writing basic writing rule of each Chinese character is learned based on a first-stage training result of a diffusion model, and structure semantics, stroke details and style semantics of each Chinese character in a hair body are learned based on a second-stage training result, so that a hair body generation model can output high-quality hair body Chinese characters.

Description

Technical Field

[0001] This application relates to the field of image generation, and in particular to a method and apparatus for generating hair based on a diffusion model. Background Technology

[0002] In the digital age, digital calligraphy, due to its convenience and ease of use, offers more and faster opportunities for calligraphy learning. Specialized research on Mao Zedong's calligraphy style will significantly promote the dissemination of calligraphy culture. Current mainstream font generation methods primarily target printed and standardized handwritten fonts, and their ability to learn styles and extract features from cursive script, with its dynamic style and open structure, is insufficient. Furthermore, Mao Zedong's calligraphy style, as a unique cursive script with immense visual tension and artistic expression, presents an even greater challenge in its generation, and related research remains largely unexplored.

[0003] Existing methods generally adopt the "content-style" paradigm in font generation, but in the task of generating highly free cursive script such as Mao's script, they still face the following defects: (1) The "content-based-style-injection" paradigm relies on the input content to regulate the generation process. This performs well in regular fonts, but it solidifies the generated character structure and makes it difficult to learn the bold and unrestrained spirit and magnificent composition of Mao's script; (2) The artistic expression of Mao's script is not only in the texture style, but also relies on structural changes, stroke rhythm and other features. However, existing methods mostly stay at the shallow texture modeling, making it difficult to capture its deep writing rules. The generated results lack the necessary expressiveness and aesthetic personality.

[0004] Existing font generation models struggle to capture the structural deformations and brushstroke details of Mao Zedong's calligraphy, resulting in a lack of individuality and expressiveness in the generated characters, making it difficult to generate high-quality Mao Zedong-style Chinese characters. Summary of the Invention

[0005] This application provides a method and apparatus for generating Mao-style Chinese characters based on a diffusion model, with the aim of generating high-quality Mao-style Chinese characters.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] A hair generation method based on a diffusion model includes:

[0008] Obtain the encoding vector and semantic information of each Chinese character in the character set; the semantic information is used to characterize the high-dimensional semantics of the Chinese character in the feature space of Mao Zedong's calligraphy style.

[0009] Using a set of calligraphy images in general cursive script as training samples, and combining the encoding vector of each Chinese character as a priori condition, a diffusion model is trained to obtain the first-stage training result of the diffusion model; the first-stage training result is used to learn the basic cursive writing rules of each Chinese character.

[0010] Based on the training results of the first stage, the diffusion model is trained using a set of calligraphy images based on Mao Zedong's style as training samples and the semantic information of each Chinese character as a priori condition, to obtain the second stage training results of the diffusion model; the second stage training results are used to learn the structural semantics, brushstroke details and stylistic semantics of each Chinese character in Mao Zedong's style.

[0011] Based on the training results of the second stage, a Mao-style font generation model is generated; the Mao-style font generation model is used to output the corresponding target Mao-style font according to the target Chinese character input by the user.

[0012] A hair generation device based on a diffusion model, comprising:

[0013] A semantic information collection unit is used to obtain the encoding vector and semantic information of each Chinese character in the character set; the semantic information is used to characterize the high-dimensional semantics of the Chinese character in the feature space of Mao Zedong's calligraphy style.

[0014] The first-stage training unit is used to train the diffusion model based on a set of general cursive calligraphy images as training samples, combined with the encoding vector of each Chinese character as a prior condition, to obtain the first-stage training result of the diffusion model; the first-stage training result is used to learn the basic cursive writing rules of each Chinese character.

[0015] The second-stage training unit is used to train the diffusion model based on the training results of the first stage, using a set of calligraphy images of Mao Zedong's style as training samples and combining the semantic information of each Chinese character as a priori conditions, so as to obtain the second-stage training results of the diffusion model; the second-stage training results are used to learn the structural semantics, brushstroke details and stylistic semantics of each Chinese character in Mao Zedong's style.

[0016] The Mao-style font generation unit is used to generate a Mao-style font generation model based on the training results of the second stage; the Mao-style font generation model is used to output the corresponding target Mao-style font according to the target Chinese character input by the user.

[0017] A storage medium comprising a stored program, wherein the program is executed by a processor to perform the diffusion-based hair generation method.

[0018] An electronic device includes: a processor, a memory, and a bus; the processor and the memory are connected via the bus.

[0019] The memory is used to store the program, and the processor is used to run the program, wherein the program is executed by the processor to perform the hair generation method based on the diffusion model.

[0020] The technical solution provided in this application obtains the encoding vector and semantic information of each Chinese character in the character set. Using a general cursive script calligraphy image set as training samples, and combining the encoding vector of each Chinese character as a prior condition, a diffusion model is trained to obtain the first-stage training result of the diffusion model. Based on the first-stage training result, using a Mao-style calligraphy image set as training samples, and combining the semantic information of each Chinese character as a prior condition, the diffusion model is trained to obtain the second-stage training result of the diffusion model. Based on the second-stage training result, a Mao-style character generation model is generated. This application learns the basic cursive writing rules of each Chinese character based on the first-stage training result of the diffusion model, and learns the structural semantics, stroke details, and stylistic semantics of each Chinese character in Mao-style characters based on the second-stage training result, enabling the Mao-style character generation model to output high-quality Mao-style Chinese characters. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic flowchart illustrating a hair generation method based on a diffusion model provided in this application embodiment;

[0023] Figure 2 A schematic flowchart of another hair generation method based on a diffusion model provided in this application embodiment;

[0024] Figure 3 A schematic flowchart of another hair generation method based on a diffusion model provided in this application embodiment;

[0025] Figure 4 A schematic diagram of the architecture of a hair generation device based on a diffusion model provided in an embodiment of this application;

[0026] Figure 5a A schematic diagram illustrating the mechanism of semantic information provided in an embodiment of this application;

[0027] Figure 5b A schematic diagram illustrating another mechanism of semantic information provided in the embodiments of this application;

[0028] Figure 6 A semantic encoding logic diagram provided for an embodiment of this application;

[0029] Figure 7 The output effect of a hair generation model provided in the embodiments of this application;

[0030] Figure 8 This is a logical diagram illustrating an overall model training scheme provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0033] like Figure 1 The diagram shown is a flowchart of a hair generation method based on a diffusion model provided in an embodiment of this application, including the following steps.

[0034] S101: Obtain the encoding vector and semantic information of each Chinese character in the character set.

[0035] Among them, semantic information is used to characterize the high-dimensional semantics of Chinese characters in the feature space of Mao's calligraphy.

[0036] Optionally, the process of obtaining the encoding vector and semantic information of each Chinese character in the character set can be found in [reference needed]. Figure 2 The steps are shown.

[0037] S102: Using a set of calligraphy images in general cursive script as training samples, and combining the encoding vector of each Chinese character as a priori condition, a diffusion model is trained to obtain the first-stage training results of the diffusion model.

[0038] The results of the first phase of training were used to learn the basic cursive writing rules for each Chinese character.

[0039] It should be noted that the training results of the first stage can be regarded as a general cursive script generation model. The training results of the first stage are based on a set of general cursive script calligraphy images. Combined with basic semantic information (i.e., the encoding vector of each Chinese character as a priori condition), the overall structure, stroke direction and style outline of cursive script characters are modeled. The aim is to generate preliminary character images with a general cursive script style, providing a structural foundation for subsequent Mao-style modeling.

[0040] In some examples, the diffusion model can be the UNet model.

[0041] In a possible implementation, the general cursive calligraphy image set can be preprocessed before use to improve its quality. Specifically, the preprocessing includes, but is not limited to, conventional operations such as image grayscale processing and font edge smoothing.

[0042] S103: Based on the training results of the first stage, using a set of calligraphy images of Mao Zedong's style as training samples and combining the semantic information of each Chinese character as a priori conditions, train the diffusion model to obtain the training results of the second stage of the diffusion model.

[0043] The results of the second phase of training were used to learn the structural semantics, brushstroke details, and stylistic semantics of each Chinese character in Mao Zedong's calligraphy style.

[0044] It should be noted that the second stage training results are the training stage of the dedicated Mao-style calligraphy generation model. Based on the first stage training results, multi-level Mao-style semantic vectors (i.e., semantic information of Chinese characters) are introduced as style guidance to achieve high-quality Mao-style Chinese character generation with distinctive style and consistent semantics, and to synergistically improve the structural rationality and style expressiveness of the generation effect.

[0045] In a possible implementation, the calligraphy image set of Mao Zedong's style can be preprocessed before using it to improve the quality of the general cursive calligraphy image set. Specifically, the preprocessing includes, but is not limited to, conventional operations such as image grayscale processing and font edge smoothing.

[0046] Optionally, the loss function used during the first-stage training results includes the mean squared error loss function, and the loss function used during the second-stage training results includes the multi-objective loss function.

[0047] In some examples, during the first-stage training, the mean square error (MES) loss function is used to supervise the learning of the predicted noise in order to optimize the denoising performance of the Unet model and thus achieve the initial generation of cursive characters. The MES loss function is calculated based on the difference between the predicted noise and the real noise.

[0048] Optionally, the multi-objective loss function includes a style contrast loss function, a wavelet decomposition loss function, and a radical-aware gating loss function; wherein, the style contrast loss function is used to estimate the style semantic difference between the generated image and the reference image; the wavelet decomposition loss function is used to estimate the stroke detail difference between the generated image and the reference image; and the radical-aware gating loss function is used to estimate the structural semantic difference between the generated image and the reference image; the generated image is the image output during the training of the diffusion model, and the reference image is the image input during the training of the diffusion model.

[0049] In some examples, the style contrast loss function is used to constrain the style features (i.e., style semantic vectors) of the generated image to be consistent with the style features of the reference image, while maintaining relative distinguishability in style features from other non-reference image samples.

[0050] In a possible implementation, style features are extracted using a pre-trained ResNet network, the similarity between feature vectors is measured using a cosine similarity function S(·), and a temperature coefficient is introduced. (For example, set to 0.9) to adjust the discriminative power of the feature distribution. Specifically, the expression of the style contrast loss function can be found in formula (13).

[0051] (13).

[0052] In formula (13), The output of the style contrast loss function is representative. Represents the generated image. Represents a reference image, f s K represents the ResNet network, and K represents the number of images with other calligraphic styles.

[0053] In some examples, the wavelet decomposition loss function is used to measure the differences in glyph structure between the generated image and the reference image.

[0054] In a possible implementation, the wavelet decomposition loss function extracts multi-scale features of the image through a wavelet attention mechanism, and is based on each layer of wavelet decomposition. The coefficients are calculated using the L1 norm to determine the differences in character shapes at different scales. Specifically, the expression for the wavelet decomposition loss function can be found in formula (14).

[0055] (14).

[0056] In formula (14), The output of the wavelet decomposition loss function is represented by L, which is a positive integer and represents the maximum value of l. l represents the index layer of the wavelet decomposition (e.g., the first wavelet decomposition, the Lth wavelet decomposition).

[0057] In some examples, the radical-aware gating loss function is used to enhance the accuracy of representation of key regions (such as radicals, components, etc.) in the generated image.

[0058] In a possible implementation, the radical perception gate loss function calculates the centroid position of the image based on image moments and divides the image into a 3×3 grid accordingly. An attention mask is generated by combining the pixel weights of each grid region to guide the diffusion model to focus on the region with a higher visual centroid. Specifically, the expression of the radical perception gate loss function can be found in formula (15).

[0059] (15).

[0060] In formula (15), represents the output of the radical perception gating loss function, and mask represents the attention mask (which can also be understood as the weight matrix). Represents the i-th generated image. This represents the i-th reference image.

[0061] It should be noted that the multi-objective loss function is a weighted sum of the style contrast loss function, wavelet decomposition loss function, radical perception gating loss function and mean square error loss function, which is used to jointly optimize the generation quality and semantic control capability of the diffusion model. Specifically, the expression of the multi-objective loss function can be found in formula (16).

[0062] (16).

[0063] In formula (16), Represents the total loss of the multi-objective loss function. Represents the preset weight. This represents the mean squared error loss function.

[0064] In some examples, the expression for the mean squared error loss function can be found in Equation (17).

[0065] (17).

[0066] In formula (17), The output represents the mean squared error loss function. This represents real noise (real Gaussian noise added during the forward process). Represents prediction noise, This represents the noise image at time t, where t represents the time step. Represents a structural semantic vector. Represents the vector of brushstroke details. Representative style semantic vector.

[0067] S104: Generate a hair generation model based on the training results of the second stage.

[0068] Among them, the Mao-style font generation model is used to output the corresponding target Mao-style font based on the target Chinese character input by the user.

[0069] In some examples, the output of the hair generation model obtained after the first and second stages of training can be seen in [reference needed]. Figure 7 As shown.

[0070] In some examples, the overall model training scheme is illustrated by a logical diagram of the two stages of training: data preparation (i.e., collecting calligraphy image sets of Mao's style and general cursive script, images of Mao's calligraphy characters, etc.), image data preprocessing, encoding embedding (i.e., performing encoding using embedding mapping functions or embedding layers), and diffusion model training. See [link to relevant documentation]. Figure 8 As shown.

[0071] In a possible implementation, the general cursive and Mao-style calligraphy image data (including calligraphy images containing multiple characters) collected during the data preparation phase can be found in Table 1.

[0072] Table 1

[0073]

[0074] The content shown in Table 1 is for illustrative purposes only.

[0075] Combination Figure 2 and Figure 3 The method described in this application addresses the problem of rigidity in structural modeling within the "content-based-style-injection" paradigm by proposing a two-stage generation strategy of "general-specific". By combining general cursive script pre-training with Mao-style fine-tuning, the diffusion model achieves stronger structural expression and detail processing capabilities while maintaining the uniformity of Mao-style calligraphy. Furthermore, to overcome the limitations of traditional "style-injection" methods that only model surface texture and stroke thickness, this paper defines "character semantics" as a multi-layered calligraphic expression structure for the first time and constructs a semantic guidance mechanism (i.e., using semantic information as a priori conditions) to provide generation conditions from multiple levels, including the overall structure, significantly enhancing the controllability and expressiveness of Mao-style calligraphy.

[0076] The process shown in S101-S104 above learns the basic cursive writing rules of each Chinese character based on the first-stage training results of the diffusion model, and learns the structural semantics, brushstroke details and stylistic semantics of each Chinese character in the Mao style based on the second-stage training results, so that the Mao style generation model can output high-quality Mao style Chinese characters.

[0077] like Figure 2 The diagram shown is a flowchart of another hair generation method based on a diffusion model provided in this application, which includes the following steps.

[0078] S201: Perform one-hot encoding on each Chinese character in the character set to obtain the one-hot vector of each Chinese character.

[0079] In this process, the Chinese characters in the character library can be numbered in advance, and then each Chinese character can be uniquely encoded.

[0080] In some examples, see formulas (1) and (2). This represents a single Chinese character, where N is the total number of Chinese characters in the character set. Each Chinese character can be represented as a one-hot vector. .

[0081] (1).

[0082] (2).

[0083] S202: Using the embedding mapping function, the one-hot vector of each Chinese character is mapped to an embedding vector in a continuous vector space.

[0084] The embedding mapping function can be pre-designed as an embedding layer for easy calling. In addition, the one-hot vector of Chinese characters is usually a discrete vector. Therefore, the one-hot vector of each Chinese character needs to be mapped to an embedding vector in a continuous vector space.

[0085] In some examples, the expressions for the embedding mapping function and the embedding vector can be found in formulas (3) and (4), specifically, This represents the embedding mapping function, used to embed one-hot vectors. Mapped to embedded vectors in a continuous vector space .

[0086] (3).

[0087] (4).

[0088] S203: Determine the encoding vector for each Chinese character based on the embedding vector corresponding to each Chinese character.

[0089] In this process, an encoding vector for each Chinese character is determined based on the embedding vector corresponding to each Chinese character. This encoding vector can be used as the training input for the diffusion model.

[0090] S204: Take each Chinese character and its corresponding Mao-style calligraphy image as input to a pre-built semantic coding model, perform semantic coding on each Mao-style calligraphy image through the semantic coding model, and output the semantic information of each Chinese character.

[0091] The semantic information includes structural semantic vectors, brushstroke detail vectors, and style semantic vectors of Chinese characters in the feature space of Mao Zedong's calligraphy style.

[0092] It should be noted that the Mao-style calligraphy images are images of Chinese characters written in Mao's style, which can be collected from the Mao-style calligraphy image dataset.

[0093] In some examples, the Mao-style calligraphy images can be preprocessed before use to improve their quality. Specifically, preprocessing includes, but is not limited to, conventional operations such as image grayscale processing and font edge smoothing.

[0094] In a possible implementation, the semantic encoding model can be regarded as a semantic extraction module, used to extract the semantic information of Chinese characters from Chinese characters and their corresponding Mao Zedong calligraphy images.

[0095] Previously, both printed and handwritten methods relied on a "content-based, style-injected" paradigm to generate images. This approach provides a content image as the foundation, with the style image influencing the content image. While "style" injection is characterized by its ease of extracting single-level information such as surface texture and stroke thickness, and its ability to quickly and stably generate glyphs, it is insufficient to express diverse information such as complete overall style, local shape, and details. To address this issue, it is urgent to introduce a special representation with multi-level and computable properties.

[0096] "Semantics" originates from linguistics and refers to the meaning of linguistic units in context. In natural language processing, semantics usually refers to the meaning of words, sentences, and contextual relationships expressed in text; in computer vision, semantics focuses on abstract information such as the semantic categories and structural relationships of objects and scenes in images; while in font generation tasks, semantics should include multi-layered font image information, paying as much attention as possible to the overall style, glyph structure, and stroke features of the target font image. Based on this, we innovatively propose the "semantics" of characters.

[0097] Wang Xizhi's theory of brushwork constructs a multi-layered system for calligraphic creation, encompassing structural norms, dynamic expression, textural communication, and individual sublimation. In his book, *Brushwork, Composition, and Character Structure—A Study of Calligraphic Morphology*, Qiu Zhenzhong analyzes the art from a visual perspective, focusing on stroke relationships, single-character structure, and overall layout. Both approaches highlight the multi-layered expressive characteristics of calligraphic art, providing a traditional aesthetic basis for the semantic layering of text images. Unlike the "style injection" method, which merely focuses on stylistic representation, this application defines the semantics of Chinese characters as a high-dimensional vector representation (i.e., high-dimensional semantics) used to characterize calligraphic features. It comprises multi-layered calligraphic information, including the overall style, basic structure, and brushstroke details of the Chinese character calligraphy image. This allows for the differentiation of different calligraphic styles and character forms, and can be used to guide font image generation tasks.

[0098] In some examples, the semantic information of Chinese characters can be represented as shown in formula (5).

[0099] (5).

[0100] In formula (5), This represents the semantic information of the Chinese character 'w' under the 's' style, and it is composed of calligraphic information vectors at different levels. This represents the overall semantic vector of the Chinese character w under the 's' style. This represents the structural semantic vector of the Chinese character w under the s style. This represents the detailed semantic vector of the Chinese character w in style s. These represent elements in the overall semantic vector, for example =1.232, then the thickness of the stroke is considered to be 1.232. These represent the elements in the structural semantic vector. These represent elements in the semantic vector of the detail layer. Specifically, the s style can be regarded as the Mao style.

[0101] In a possible implementation, the embodiments of the present application construct a well-structured "semantics of Chinese characters" representation system, which consists of three interrelated levels: the overall level is the a vector, which focuses on the overall style and visual layout of the Chinese characters, covering the writing center of gravity, the trend of the line of characters, and the coherence of the brushwork, ensuring the overall style and consistency; the structural semantics b vector does not correspond to a fixed structural layout, but abstractly models the "dynamic characteristics of the configuration layout" in the calligraphy font, such as the vertical and horizontal stagger, the relationship between the main and secondary parts, the center of gravity distribution, and the spatial tension between the strokes; the c vector represents the detail level, by focusing on the brushwork performance and local dynamic changes, such as the starting and ending ways of the brush, the thickness of the brush tip, and the turning strength, enhancing the local expressiveness and detail restoration. Through this multi-level structure, the semantics of Chinese characters can comprehensively reflect calligraphy information at different scales, providing strong support for the accurate modeling and generation of complex Mao-style fonts. It should be noted that the above two fonts are displayed under the semantics of the Mao style, and when facing the semantics of other calligraphers, the semantic vector content of the same Chinese character changes. And because it is too cumbersome and subjective to manually extract the semantics of Chinese characters, the semantics of Mao-style Chinese characters can be automatically extracted through a neural network.

[0102] In a possible implementation, the action mechanism of semantic information can be manifested as Figure 5a and Figure 5b shown, where Figure 5a is an example of semantic splitting of the Mao-style Chinese character "zong", Figure 5b is an example of semantic splitting of the Mao-style Chinese character "cai". Specifically, compared with the traditional "style injection" method that only captures font features in a single dimension such as surface texture, the introduction of semantic information significantly expands the breadth and depth of information expression. By dividing the Chinese character image into multiple levels such as overall style, local structure, and detailed brushwork, the semantic representation can more systematically describe the internal configuration rules and visual characteristics of the font.

[0103] Optionally, the semantic encoding model includes an embedding layer, a gated attention mechanism, a wavelet attention mechanism, a style encoding module, and a cross-attention mechanism; among them, the wavelet attention mechanism includes a Haar wavelet decomposition layer, a residual network convolutional layer, a convolutional block attention (Convolutional Block Attention Module, CBMA) module, and a global average pooling layer; the style encoding module includes an encoder pre-trained on multiple calligraphy works.

[0104] In some examples, the encoder in the style encoding module includes a ResNet-18 model, and the output layer of the ResNet-18 model is replaced by a classification layer. The ResNet-18 model is a deep convolutional neural network model implemented by stacking residual blocks (Basic Block).

[0105] In some examples, the Haar wavelet decomposition layer in the wavelet attention mechanism is a single layer, and the residual network convolutional layer includes four layers of cross-linked ResNet (Residual Network) convolutional layers.

[0106] Optionally, the implementation process of semantic encoding of Mao Zedong's calligraphy images using the semantic encoding model can be found in [link to documentation]. Figure 3 The steps are shown.

[0107] The processes described in S201-S204 above can obtain the encoding vector and semantic information of each Chinese character in the character library, providing valuable reference data for the subsequent training of the diffusion model.

[0108] like Figure 3 The diagram shown is a flowchart of another hair generation method based on a diffusion model provided in this application, which includes the following steps.

[0109] S301: Encode Chinese characters through an embedding layer to obtain the corresponding feature vectors.

[0110] The embedding layer includes preset embedding mapping functions.

[0111] S302: By using a gating attention mechanism, the feature vector is decomposed to obtain the first feature component and the second feature component, and the first feature component and the second feature component are fused to obtain the structural semantic vector of Chinese characters in the feature space of Mao Zedong's calligraphy.

[0112] Among them, the eigenvector Fe is decomposed into the first eigencomponent. With the second characteristic component The two are added together to form a structural semantic vector. The aim is to guide the diffusion model to focus on important structures such as radicals and components, thereby extracting structural semantics.

[0113] In some examples, the expression for the structural semantic vector can be found in formula (6).

[0114] (6).

[0115] In formula (6), This represents the preset weight.

[0116] S303: Perform discrete wavelet transform on the Mao-style calligraphy image corresponding to the Chinese character to obtain low-frequency and high-frequency components.

[0117] Among them, Mao-style calligraphy images Typically represented as a tensor of size H×W with 1 channel, the Mao-style calligraphy image undergoes Discrete Wavelet Transform (DWT) processing to extract low-frequency components. With high frequency components .

[0118] S304: The low-frequency component and the high-frequency component are fused to obtain the corresponding feature representation.

[0119] Among them, the low-frequency components With high frequency components By performing fusion (e.g., addition), the corresponding feature representations are obtained. .

[0120] In some examples, discrete wavelet transform is performed on the Mao-style characters corresponding to Chinese characters to obtain low-frequency and high-frequency components. The low-frequency and high-frequency components are then fused to obtain the corresponding feature representation. This process can be summarized as shown in formulas (7) and (8).

[0121] (7).

[0122] (8).

[0123] S305: By using a wavelet attention mechanism, feature representation is enhanced to obtain the stroke detail vector of Chinese characters in the feature space of Mao Zedong's calligraphy.

[0124] Specifically, the wavelet attention mechanism performs channel compression through a 1×1 convolutional layer, adjusting the number of channels to 64, to obtain the feature tensor. Then, the CBAM module is used to generate detailed vectors of the brushstrokes. .

[0125] In some examples, the feature representation is enhanced by wavelet attention mechanism to obtain the implementation process of the stroke detail vector of Chinese characters in the feature space of Mao's calligraphy. See formulas (9) and (10).

[0126] (9).

[0127] (10).

[0128] S306: Through the encoder, style extraction is performed on the Mao-style calligraphy images corresponding to the Chinese characters to obtain the style semantic vector of the Chinese characters in the feature space of Mao-style calligraphy.

[0129] The encoder maps images of Mao-style characters to the calligraphy style in the style space that is closest to the target character shape, and finally extracts the style semantic vector. 。

[0130] In some examples, the encoder is denoted as , and the process of extracting the style semantic vector can be denoted as shown in formula (11).

[0131] (11).

[0132] In formula (11), represents the cursive script image.

[0133] S307: Through the cross-attention mechanism, fuse the structural semantic vector, pen gesture detail vector, and style semantic vector of Chinese characters to obtain the corresponding target semantic vector.

[0134] Among them, the process of fusing the structural semantic vector, pen gesture detail vector, and style semantic vector of Chinese characters to obtain the corresponding target semantic vector can be seen as shown in formula (12).

[0135] (12).

[0136] In formula (12), represents the target semantic vector, represents the structural semantic vector, represents the pen gesture detail vector, represents the style semantic vector.

[0137] S308: Based on the target semantic vector, output the semantic information of Chinese characters.

[0138] In some examples, referring to the semantic encoding logic schematic diagram shown in Figure 6 , the process of semantic encoding can be understood as follows: Taking the character "你" as an example, first, this character passes through the embedding layer encoding to obtain a feature vector; subsequently, through the gated attention mechanism, the feature vector is decomposed into a first feature component and a second feature component, and the two are added to form a structural semantic vector, aiming to guide the model to focus on important structures such as radicals and components, so as to extract detailed semantics; The cursive script image is used as a reference image, and first undergoes discrete wavelet transform to extract the low-frequency component and the high-frequency component; After fusing the low-frequency component and the high-frequency component, to enhance the adaptive channel fusion ability of this feature, first use a convolutional layer with a convolution kernel of 1×1 to adjust the number of channels to 64, and then perform channel attention and spatial attention calculations to obtain the pen gesture detail vector; Use an encoder pre-trained on various calligraphy works to map the reference image to the calligraphy style closest to the target glyph in the style space, and finally extract the style semantic vector; The three-layer semantic vectors are fused into a target semantic vector through the cross-attention mechanism.

[0139] It should be noted that the semantic information of Chinese characters theoretically expands the representation space of font modeling and provides a feasible implementation path from a technical perspective. This mechanism makes up for the shortcomings of existing methods in style modeling and provides a structured and controllable expressive foundation for modeling complex calligraphic styles.

[0140] The processes described in S301-S308 above can effectively achieve semantic encoding of Mao Zedong's calligraphy images.

[0141] like Figure 4 The diagram shown is a schematic representation of the architecture of a hair generation device based on a diffusion model provided in an embodiment of this application, including the following units.

[0142] The semantic information collection unit 100 is used to obtain the encoding vector and semantic information of each Chinese character in the character library; the semantic information is used to characterize the high-dimensional semantics of Chinese characters in the feature space of Mao Zedong's calligraphy.

[0143] Optionally, the semantic information collection unit 100 is specifically used for: performing one-hot encoding on each Chinese character in the character library to obtain the one-hot vector of each Chinese character; using an embedding mapping function to map the one-hot vector of each Chinese character into an embedding vector in a continuous vector space; determining the encoding vector of each Chinese character based on the embedding vector corresponding to each Chinese character; taking each Chinese character and its corresponding Mao-style calligraphy image as input to a pre-constructed semantic encoding model, performing semantic encoding on each Mao-style calligraphy image through the semantic encoding model, and outputting the semantic information of each Chinese character; the semantic information includes the structural semantic vector, stroke detail vector, and style semantic vector of the Chinese character in the feature space of Mao-style calligraphy.

[0144] Optionally, the semantic encoding model includes an embedding layer, a gated attention mechanism, a wavelet attention mechanism, a style encoding module, and a cross-attention mechanism; among which, the wavelet attention mechanism includes a Haar wavelet decomposition layer, a residual network convolutional layer, a convolutional block attention module, and a global average pooling layer; the style encoding module includes an encoder pre-trained on various calligraphy works.

[0145] Optionally, the semantic encoding process includes: encoding Chinese characters through an embedding layer to obtain corresponding feature vectors; decomposing the feature vectors through a gating attention mechanism to obtain first and second feature components, and fusing the first and second feature components to obtain the structural semantic vector of the Chinese character in the feature space of Mao Zedong's calligraphy; performing discrete wavelet transform on the Mao Zedong calligraphy image corresponding to the Chinese character to obtain low-frequency and high-frequency components; fusing the low-frequency and high-frequency components to obtain the corresponding feature representation; performing feature enhancement on the feature representation through a wavelet attention mechanism to obtain the stroke detail vector of the Chinese character in the feature space of Mao Zedong's calligraphy; extracting the style of the Mao Zedong calligraphy image corresponding to the Chinese character through an encoder to obtain the style semantic vector of the Chinese character in the feature space of Mao Zedong's calligraphy; fusing the structural semantic vector, stroke detail vector, and style semantic vector of the Chinese character through a cross-attention mechanism to obtain the corresponding target semantic vector; and outputting the semantic information of the Chinese character based on the target semantic vector.

[0146] The first-stage training unit 200 is used to train the diffusion model based on a set of calligraphy images in general cursive script as training samples, combined with the encoding vector of each Chinese character as a prior condition, so as to obtain the first-stage training results of the diffusion model; the first-stage training results are used to learn the basic cursive writing rules of each Chinese character.

[0147] The second-stage training unit 300 is used to train the diffusion model based on the first-stage training results, using a set of calligraphy images of Mao Zedong's style as training samples and combining the semantic information of each Chinese character as a priori conditions, so as to obtain the second-stage training results of the diffusion model. The second-stage training results are used to learn the structural semantics, brushstroke details and stylistic semantics of each Chinese character in Mao Zedong's style.

[0148] Optionally, the loss function used during the first-stage training results includes the mean squared error loss function, and the loss function used during the second-stage training results includes the multi-objective loss function.

[0149] Optionally, the multi-objective loss function includes a style contrast loss function, a wavelet decomposition loss function, and a radical-aware gating loss function; wherein, the style contrast loss function is used to estimate the style semantic difference between the generated image and the reference image; the wavelet decomposition loss function is used to estimate the stroke detail difference between the generated image and the reference image; and the radical-aware gating loss function is used to estimate the structural semantic difference between the generated image and the reference image; the generated image is the image output during the training of the diffusion model, and the reference image is the image input during the training of the diffusion model.

[0150] The Mao-style font generation unit 400 is used to generate a Mao-style font generation model based on the training results of the second stage; the Mao-style font generation model is used to output the corresponding target Mao-style font according to the target Chinese character input by the user.

[0151] Each unit shown above learns the basic cursive writing rules of each Chinese character based on the first-stage training results of the diffusion model, and learns the structural semantics, brushstroke details, and stylistic semantics of each Chinese character in the Mao style based on the second-stage training results, so that the Mao style generation model can output high-quality Mao style Chinese characters.

[0152] This application also provides a computer-readable storage medium including a stored program, wherein the program executes the hair generation method based on the diffusion model provided in this application.

[0153] This application also provides an electronic device, including a processor, a memory, and a bus. The processor and the memory are connected via the bus. The memory is used to store a program, and the processor is used to run the program. During program execution, the diffusion-based hair generation method provided in this application is implemented.

[0154] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0155] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A hair generation method based on a diffusion model, characterized in that, include: Obtain the encoding vector and semantic information of each Chinese character in the character set; the semantic information is used to characterize the high-dimensional semantics of the Chinese character in the feature space of Mao Zedong's calligraphy style. Using a set of calligraphy images in general cursive script as training samples, and combining the encoding vector of each Chinese character as a priori condition, a diffusion model is trained to obtain the first-stage training result of the diffusion model. The results of the first phase of training are used to learn the basic cursive writing rules for each of the Chinese characters. Based on the training results of the first stage, the diffusion model is trained using a set of calligraphy images of Mao Zedong's style as training samples and the semantic information of each Chinese character as a priori condition, so as to obtain the second stage training results of the diffusion model. The results of the second phase of training are used to learn the structural semantics, stroke details, and stylistic semantics of each Chinese character in the Mao style. Based on the training results of the second stage, a Mao-style font generation model is generated; the Mao-style font generation model is used to output the corresponding target Mao-style font according to the target Chinese character input by the user.

2. The method according to claim 1, characterized in that, Obtain the encoding vector and semantic information of each Chinese character in the character set, including: Each Chinese character in the character set is encoded using one-hot encoding to obtain a one-hot vector for each Chinese character. Using an embedding mapping function, the one-hot vector of each Chinese character is mapped to an embedding vector in a continuous vector space; Based on the embedding vector corresponding to each Chinese character, the encoding vector for each Chinese character is determined; Each Chinese character and its corresponding Mao-style calligraphy image are used as input to a pre-constructed semantic encoding model. The semantic encoding model performs semantic encoding on each Mao-style calligraphy image and outputs the semantic information of each Chinese character. The semantic information includes the structural semantic vector, stroke detail vector, and style semantic vector of the Chinese character in the feature space of Mao-style calligraphy.

3. The method according to claim 2, characterized in that, The semantic encoding model includes an embedding layer, a gated attention mechanism, a wavelet attention mechanism, a style encoding module, and a cross-attention mechanism; wherein, the wavelet attention mechanism includes a Haar wavelet decomposition layer, a residual network convolutional layer, a convolutional block attention module, and a global average pooling layer; the style encoding module includes an encoder pre-trained on a variety of calligraphy works.

4. The method according to claim 3, characterized in that, The semantic encoding process includes: The Chinese characters are encoded through the embedding layer to obtain the corresponding feature vectors; The feature vector is decomposed through the gating attention mechanism to obtain a first feature component and a second feature component, and the first feature component and the second feature component are fused to obtain the structural semantic vector of the Chinese character in the feature space of Mao Zedong's calligraphy. The discrete wavelet transform is performed on the Mao-style calligraphy image corresponding to the Chinese character to obtain the low-frequency component and the high-frequency component. The low-frequency component and the high-frequency component are fused to obtain the corresponding feature representation; The wavelet attention mechanism is used to enhance the feature representation to obtain the stroke detail vector of the Chinese character in the feature space of Mao's calligraphy. The encoder is used to extract the style of the Mao-style calligraphy image corresponding to the Chinese character, so as to obtain the style semantic vector of the Chinese character in the feature space of Mao-style calligraphy. The cross-attention mechanism is used to fuse the structural semantic vector, stroke detail vector, and style semantic vector of the Chinese character to obtain the corresponding target semantic vector. Based on the target semantic vector, the semantic information of the Chinese character is output.

5. The method according to claim 1, characterized in that, The loss function used during the first stage of training includes the mean squared error loss function, while the loss function used during the second stage of training includes the multi-objective loss function.

6. The method according to claim 5, characterized in that, The multi-objective loss function includes a style contrast loss function, a wavelet decomposition loss function, and a radical-aware gating loss function; wherein, the style contrast loss function is used to estimate the style semantic difference between the generated image and the reference image; the wavelet decomposition loss function is used to estimate the stroke detail difference between the generated image and the reference image; the radical-aware gating loss function is used to estimate the structural semantic difference between the generated image and the reference image; the generated image is the image output during the training of the diffusion model, and the reference image is the image input during the training of the diffusion model.

7. A hair generation device based on a diffusion model, characterized in that, include: The semantic information collection unit is used to obtain the encoding vector and semantic information of each Chinese character in the character set; The semantic information is used to characterize the high-dimensional semantics of the Chinese characters in the feature space of Mao Zedong's calligraphy style; The first-stage training unit is used to train the diffusion model based on a set of calligraphy images in general cursive script as training samples, combined with the encoding vector of each Chinese character as a prior condition, so as to obtain the first-stage training result of the diffusion model. The results of the first phase of training are used to learn the basic cursive writing rules for each of the Chinese characters. The second-stage training unit is used to train the diffusion model based on the training results of the first stage, using a set of calligraphy images of Mao Zedong's style as training samples and combining the semantic information of each Chinese character as a priori conditions, so as to obtain the second-stage training results of the diffusion model. The results of the second phase of training are used to learn the structural semantics, stroke details, and stylistic semantics of each Chinese character in the Mao style. The Mao-style font generation unit is used to generate a Mao-style font generation model based on the training results of the second stage; the Mao-style font generation model is used to output the corresponding target Mao-style font according to the target Chinese character input by the user.

8. The apparatus according to claim 7, characterized in that, The semantic information collection unit is specifically used for: Each Chinese character in the character set is encoded using one-hot encoding to obtain a one-hot vector for each Chinese character. Using an embedding mapping function, the one-hot vector of each Chinese character is mapped to an embedding vector in a continuous vector space; Based on the embedding vector corresponding to each Chinese character, the encoding vector for each Chinese character is determined; Each Chinese character and its corresponding Mao-style calligraphy image are used as input to a pre-constructed semantic encoding model. The semantic encoding model performs semantic encoding on each Mao-style calligraphy image and outputs the semantic information of each Chinese character. The semantic information includes the structural semantic vector, stroke detail vector, and style semantic vector of the Chinese character in the feature space of Mao-style calligraphy.

9. A storage medium, characterized in that, The storage medium includes a stored program, wherein the program is executed by a processor to perform the hair generation method based on the diffusion model according to any one of claims 1-6.

10. An electronic device, characterized in that, include: Processor, memory, and bus; The processor and the memory are connected via the bus; The memory is used to store a program, and the processor is used to run the program, wherein the program is executed by the processor to perform the hair generation method based on the diffusion model according to any one of claims 1-6.

Images