Text image generation method and diffusion generative model training method
By combining diffusion generation model and noise data, the problems of limited materials and inefficient regular design in the prior art are solved, and high-quality and diverse text image generation, especially the generation of handwritten text images.
Patent Information
- Application Number
- PCT/SG2024/050338
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-23
- Filing Date
- 2024-05-22
- Publication Date
- 2024-11-28
AI Technical Summary
When generating high-quality text images, the collected image background material is limited, and the design and synthesis rules are inefficient and limited, resulting in the inability to generate high-quality text images, especially the inability to effectively generate handwritten text images.
Using the diffusion generation model, by obtaining the image features and noise data of the initial image, the target text image is generated based on the visual and semantic features of the sample text image. Sample noise is introduced when training the model to amplify the data, and the context information and texture features of the text image are encoded. .
It significantly improves the quality of the generated text images, and can generate diverse and high-quality text images, including handwritten text images, without the need to collect a large amount of fonts and background materials, simplifying the training and reasoning process.
Smart Images

Figure SG2024050338_28112024_PF_FP_ABST
Abstract
Description
[0001] Cross-Reference to Patent Application No. 202310606347.X, filed with the Chinese Patent Office on May 23, 2023, entitled "Text Image Generation Method and Diffusion Generative Model Training Method," the entire contents of which are incorporated herein by reference. Technical Field: Embodiments of the present disclosure relate to the field of computer technology, and more particularly, to a method for generating text images. One or more embodiments of the present disclosure also relate to a method for generating handwritten text images, a method for training a diffusion generative model, a text image generation device, a handwritten text image generation device, a diffusion generative model training device, a computing device, a computer-readable storage medium, and a computer program. Background: With the development of computer technology, the generation of text images has become increasingly dependent on computer technology. Currently, text images can typically be generated based on rules: various fonts and image background materials are collected, and a synthetic text image is rendered based on the arrangement rules of the text in the text image. However, the image background material collected by the above method is limited, and different synthesis rules must be designed for different scenarios, which is very inefficient and has limitations. This makes it impossible to synthesize high-quality text images. Therefore, a high-quality text image generation solution is urgently needed. In view of this, embodiments of the present disclosure provide a text image generation method. One or more embodiments of the present disclosure also relate to a handwritten text image generation method, a diffusion generation model training method, a text image generation device, a handwritten text image generation device, a diffusion generation model training device, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art. According to a first aspect of an embodiment of the present disclosure, a text image generation method is provided, comprising: obtaining an initial image; inputting the initial image into a conditional encoder to obtain image features of the initial image; inputting the initial image and the image features into a diffusion generation model to obtain noise data corresponding to the initial image, wherein the diffusion generation model is trained based on a sample text image, sample image features of the sample text image, and a noise sample image, the noise sample image is obtained by adding sample noise to the sample text image, and the sample image features include at least one of sample visual features and sample semantic features; and generating a target text image corresponding to the initial image based on the initial image and the noise data.According to a second aspect of an embodiment of the present disclosure, a method for generating a handwritten text image is provided, comprising: receiving a handwritten text image generation request sent by a user, wherein the handwritten text image generation request carries an initial image and initial style information of the initial image; inputting the initial image and the initial style information into a conditional encoder to obtain visual features, semantic features, and style features of the initial image; inputting the initial image, the visual features, the semantic features, and the style features into a diffusion generation model to obtain noise data corresponding to the initial image, wherein the diffusion generation model is trained based on a sample text image, sample image features of the sample text image, and a noise sample image, wherein the noise sample image is obtained by adding sample noise to the sample text image, and the sample image features include sample visual features, sample semantic features, and sample style features; generating a target handwritten text image corresponding to the initial image based on the initial image and the noise data; and sending the target handwritten text image corresponding to the initial image to the user. According to a third aspect of an embodiment of the present disclosure, a diffusion generation model training method is provided, which is applied to a cloud-side device and includes: obtaining a sample set, wherein the sample set includes multiple sample text images, the sample text images carry sample image features, and the sample image features include at least one of sample visual features and sample semantic features; extracting a first sample text image from the sample set, wherein the first sample text image is any one of the multiple sample text images; adding first sample noise to the first sample text image to obtain a first noise sample image; inputting the first noise sample image and the first sample image features carried by the first sample text image into a diffusion generation model to obtain a first predicted noise; calculating a noise loss value based on the first sample noise and the first predicted noise; adjusting model parameters of the diffusion generation model based on the noise loss value, and returning to the step of extracting the first sample text image from the sample set until a second preset stop condition is met, thereby obtaining model parameters of the trained diffusion generation model; and sending the model parameters of the diffusion generation model to the end-side device. According to a fourth aspect of an embodiment of the present disclosure, a text image generation device is provided, comprising: a first acquisition module, configured to acquire an initial image; a first input module, configured to input the initial image into a conditional encoder to obtain image features of the initial image; a second input module, configured to input the initial image and the image features into a diffusion generation model to obtain noise data corresponding to the initial image, wherein the diffusion generation model is trained based on a sample text image, sample image features of the sample text image, and a noise sample image, the noise sample image is obtained by adding sample noise to the sample text image, and the sample image features include at least one of sample visual features and sample semantic features; and a first generation module, configured to generate a target text image corresponding to the initial image based on the initial image and the noise data.According to a fifth aspect of an embodiment of the present disclosure, a handwritten text image generation apparatus is provided, comprising: a receiving module configured to receive a handwritten text image generation request sent by a user, wherein the handwritten text image generation request carries an initial image and initial style information of the initial image; a third input module configured to input the initial image and the initial style information into a conditional encoder to obtain visual features, semantic features, and style features of the initial image; a fourth input module configured to input the initial image, the visual features, the semantic features, and the style features into a diffusion generation model to obtain noise data corresponding to the initial image, wherein the diffusion generation model is trained based on a sample text image, sample image features of the sample text image, and a noise sample image, wherein the noise sample image is obtained by adding sample noise to the sample text image, and the sample image features include sample visual features, sample semantic features, and sample style features; a second generation module configured to generate a target handwritten text image corresponding to the initial image based on the initial image and the noise data; and a first sending module configured to send the target handwritten text image corresponding to the initial image to the user. According to a sixth aspect of an embodiment of the present disclosure, a diffusion generation model training apparatus is provided, which is applied to a cloud-side device and includes: a second acquisition module, configured to acquire a sample set, wherein the sample set includes multiple sample text images, the sample text images carry sample image features, and the sample image features include at least one of sample visual features and sample semantic features; an extraction module, configured to extract a first sample text image from the sample set, wherein the first sample text image is any one of the multiple sample text images; an addition module, configured to add a first sample noise to the first sample text image to obtain a first noise sample image; a fifth input module, configured to input the first noise sample image and the first sample image features carried by the first sample text image into the diffusion generation model to obtain a first predicted noise; a calculation module, configured to calculate a noise loss value based on the first sample noise and the first predicted noise; an adjustment module, configured to adjust model parameters of the diffusion generation model based on the noise loss value, and return to executing the step of extracting the first sample text image from the sample set until a second preset stop condition is reached, thereby obtaining model parameters of the trained diffusion generation model; a second sending module, The present invention is configured to send model parameters of a diffusion generation model to an end-side device. According to a seventh aspect of an embodiment of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions. When executed by the processor, the computer-executable instructions implement the steps of the method provided in the first, second, or third aspect above.According to an eighth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, storing computer-executable instructions. When executed by a processor, the instructions implement the steps of the method provided in the first, second, or third aspect. According to a ninth aspect of the embodiments of the present disclosure, a computer program is provided. When executed on a computer, the computer is caused to execute the steps of the method provided in the first, second, or third aspect. A text image generation method provided in one embodiment of the present disclosure comprises obtaining an initial image; inputting the initial image into a conditional encoder to obtain image features of the initial image; inputting the initial image and the image features into a diffusion generation model to obtain noise data corresponding to the initial image. The diffusion generation model is trained based on a sample text image, sample image features of the sample text image, and a noise sample image. The noise sample image is obtained by adding sample noise to the sample text image. The sample image features include at least one of sample visual features and sample semantic features. Based on the initial image and the noise data, a target text image corresponding to the initial image is generated. Since sample noise is introduced when training the diffusion generative model, an infinite number of noise sample images can be generated based on the sample noise, which expands the training data of the diffusion generative model. In addition, sample image features related to sample text images are also introduced when training the diffusion generative model, effectively encoding the contextual information and hierarchical features between characters in the sample text images, significantly improving the quality of the text images generated by the diffusion generative model. BRIEF DESCRIPTION OF THE DRAWINGS FIG1 is an architecture diagram of a text image generation system provided by one embodiment of the present disclosure; FIG2 is a flowchart of a text image generation method provided by one embodiment of the present disclosure; FIG3 is a process flow chart of a diffusion model training method provided by one embodiment of the present disclosure; FIG4 is a process flow chart of a text image generation method provided by one embodiment of the present disclosure; FIG5 is a flowchart of a handwritten text image generation method provided by one embodiment of the present disclosure; FIG6 is a schematic diagram of a text image generated in a handwritten scene provided by one embodiment of the present disclosure; FIG7 is a schematic diagram of a text image generated in a natural scene provided by one embodiment of the present disclosure; FIG8 is a flowchart of a diffusion generation model training method provided by one embodiment of the present disclosure; FIG9 is a schematic diagram of a text image generation device provided by one embodiment of the present disclosure; FIG10 is a schematic diagram of a handwritten text image generation device provided by one embodiment of the present disclosure; FIG11 is a schematic diagram of a diffusion generation model training device provided by one embodiment of the present disclosure; FIG12 is a block diagram of a computing device provided by one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS Exemplary embodiments will be described in detail herein, with examples shown in the accompanying drawings.When the following description refers to the accompanying drawings, unless otherwise indicated, like numbers in different drawings represent like or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are examples of apparatuses and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims. Numerous specific details are set forth in the following description to facilitate a thorough understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art may make similar generalizations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific embodiments disclosed below. The terms used in one or more embodiments of the present disclosure are for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms "a," "an," "the," and "the" used in one or more embodiments of the present disclosure and the appended claims are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should be understood that although the terms "first," "second," and so on may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are used to distinguish the same type of information from one another. For example, the first may also be referred to as the second, and similarly, the second may also be referred to as the first, without departing from the scope of one or more embodiments of the present disclosure. Depending on the context, the word "if" as used herein may be interpreted as "at the time," "when," or "in response to determining." In addition, it should be noted that user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present disclosure are not limited to these terms. , are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse. First, the terms used in one or more embodiments of this disclosure are explained. Optical Character Recognition: Optical Character Recognition (OCR) refers to the process by which an electronic device (such as a scanner or digital camera) determines the shape of a character by detecting dark and light patterns, and then uses character recognition methods to translate the character shape into computer text.Generative Adversarial Network (GAN): A generative adversarial network (GAN) is a generative model that learns through a game of two neural networks. It consists of a generator and a discriminator, and can learn generative tasks without using labeled data. The generator randomly samples from the latent space as input, and its output must closely resemble real samples. The discriminator takes real samples or the generator's output as input, and its goal is to distinguish the generator's output from real samples as much as possible. The generator and discriminator compete with each other and continuously learn, ultimately making it impossible for the discriminator to determine whether the generator's output is realistic. Contrastive Language - Image Pre-training (CLIP): The Contrastive Language - Image Pre-training (CLIP) model is a multimodal pre-training model based on parallel training of images and text.
[0002] 00V: Out of Vocabulary. In natural language processing or text processing, a vocabulary is often used. This vocabulary can be pre-loaded, custom-defined, or extracted from the current dataset. Imagine a new dataset with some words not in the existing vocabulary. These words are called out of vocabulary.
[0003] UNet: A UNet is an encoder-decoder network based on convolutional neural networks and skip connections, typically configured to generate images of the same size as the input image. With the advancement of computer technology, the generation of text images has become increasingly dependent on computer technology. Currently, text images can be generated using the following methods: Rule-based text image generation schemes: A variety of fonts and background materials are collected and rendered and synthesized based on the layout rules of the text within the text image. However, rule-based text image generation schemes have limited collection resources and require the design of different synthesis rules for different scenarios, making them inefficient and limiting. Furthermore, this scheme cannot synthesize effective handwritten text images. This is because real handwritten text is unconstrained. The same character is generally different when written by any writer, and the writing style varies significantly between writers. However, the same character rendered using handwritten fonts will be completely consistent, lacking diversity. Generative adversarial network-based text image generation approaches: These approaches treat the image generation problem as a competition between two modules: a discriminator and a generator. Specifically, the generator generates synthetic images from a given noise source, while the discriminator distinguishes between the generator's output and real images. The generator attempts to produce images that are closer to reality, while the discriminator attempts to distinguish between real and generated images. However, these approaches lack the diversity of generated samples compared to likelihood-based models. Furthermore, these approaches are often difficult to train and are prone to mode collapse without carefully selected hyperparameters and regularizers. Furthermore, these approaches trade diversity for fidelity in generated text images, resulting in the generated text images not covering the entire distribution. Based on this, embodiments of the present disclosure provide a method for generating a text image. The method comprises the following steps: obtaining an initial image; inputting the initial image into a conditional encoder to obtain image features of the initial image; and inputting the initial image and image features into a diffusion generative model to obtain noise data corresponding to the initial image. The diffusion generative model is trained based on a sample text image, sample image features of the sample text image, and a noise sample image. The noise sample image is obtained by adding sample noise to the sample text image. The sample image features include at least one of sample visual features and sample semantic features. A target text image corresponding to the initial image is generated based on the initial image and the noise data. Because sample noise is introduced during training of the diffusion generative model, an infinite number of noise sample images can be generated based on the sample noise, thus expanding the training data for the diffusion generative model. Furthermore, sample image features related to the sample text image are also introduced during training of the diffusion generative model, effectively encoding contextual information and texture features between characters in the sample text image, significantly improving the quality of text images generated by the diffusion generative model.The present disclosure provides a method for generating a text image. The present disclosure also relates to a method for generating a handwritten text image, a method for training a diffusion generation model, a text image generating apparatus, a handwritten text image generating apparatus, a diffusion generation model training apparatus, a computing device, a computer-readable storage medium, and a computer program, each of which is described in detail in the following embodiments. 1 shows an architecture diagram of a text image generation system provided by an embodiment of the present disclosure. The text image generation system may include a client 100 and a server 200; the client 100 is configured to send an initial image to the server 200; the server 200 is configured to input the initial image into a conditional encoder to obtain image features of the initial image; the initial image and the image features are input into a diffusion generation model to obtain noise data corresponding to the initial image, wherein the diffusion generation model is trained based on a sample text image, sample image features of the sample text image, and a noise sample image, wherein the noise sample image is obtained by adding sample noise to the sample text image, and the sample image features include at least one of sample visual features and sample semantic features; a target text image corresponding to the initial image is generated based on the initial image and the noise data; the target text image is sent to the client 100; and the client 100 is further configured to receive the target text image sent by the server 200. By applying the solution of the embodiments of the present disclosure, since sample noise is introduced when training the diffusion generation model, an infinite number of noise sample images can be generated based on the sample noise, thus expanding the training data for the diffusion generation model. Furthermore, sample image features related to sample text images are also introduced when training the diffusion generation model, effectively encoding the contextual information and texture features between characters in the sample text images, significantly improving the quality of text images generated by the diffusion generation model. In practical applications, the text image generation system may include multiple clients 100 and a server 200. Multiple clients 100 can establish communication connections through the server 200. In the text image generation scenario, the server 200 is used to provide text image generation services between multiple clients 100. Multiple clients 100 can act as senders or receivers, respectively, and communicate through the server 200. Users can interact with the server 200 through the client 100 to receive data from other clients 100 or send data to other clients 100. In the text image generation scenario, a user may publish a data stream to a server 200 via a client 100. The server 200 generates a text image based on the data stream and pushes the text image to other clients with which the communication is established. The connection between the client 100 and the server 200 is established via a network.The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by the client 100 may require encoding, transcoding, compression, and other processing before being released to the server 200. The client 100 can be a browser, an application (APP), a web application such as an H5 (HyperText Markup Language, Version 5) application, a light application (also known as a mini-program, a lightweight application), or a cloud application. The client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by the server 200, such as the real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and rely on the device or certain applications in the device to operate. For example, the electronic device may have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, tablet computer, or personal computer. Electronic devices can also typically be configured with various other types of applications, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like. The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers configured to provide background training to support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers or as a single server. The server can also be a server in a distributed system or a server integrated with blockchain. The server can also be a cloud server (cloud-side device) providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), big data, and artificial intelligence platforms, or an intelligent cloud computing server or intelligent server equipped with artificial intelligence technology.It is worth noting that the text image generation method provided in the embodiments of the present disclosure is generally executed by the server. However, in other embodiments of the present disclosure, the client may also have similar functions to the server to execute the text image generation method provided in the embodiments of the present disclosure. In other embodiments, the text image generation method provided in the embodiments of the present disclosure may also be jointly executed by the client and the server. Referring to FIG2 , FIG2 shows a flowchart of a text image generation method provided in one embodiment of the present disclosure, specifically comprising the following steps: Step 202: Obtaining an initial image. In one or more embodiments of the present disclosure, when generating a text image, an initial image may be obtained and used as a reference image in the text image generation process to obtain a target text image corresponding to the initial image. Specifically, the initial image refers to a reference image in the text image generation process and may be an initial text image including text content or an initial noise image carrying initial text information, wherein the initial noise image does not include text content. The selection of the initial image is determined based on actual circumstances and is not limited in the embodiments of the present disclosure. The initial image can be an image of a variety of scenes, including but not limited to handwritten text images, street scene text images, and ancient book images. The selection depends on practical application and is not limited in this embodiment. In practical applications, there are various ways to obtain the initial image, and the selection depends on practical application and is not limited in this embodiment. In one possible implementation of this disclosure, the initial image can be read from another data acquisition device or database. In another possible implementation of this disclosure, the initial image can be received from a user. Step 204: Input the initial image into a conditional encoder to obtain image features of the initial image. In one or more embodiments of this disclosure, after obtaining the initial image, the initial image can be input into the conditional encoder for processing by the conditional encoder to obtain image features of the initial image. Specifically, the image features can represent the contextual information of the text in the initial image and can also represent the texture information of the initial image. The image features of the initial image include visual features and / or semantic features. If the initial image is an image of handwritten text or contains a font style, the image features of the initial image can also include style features. Visual features represent the overall texture information of the initial image, semantic features represent the contextual semantic information between text characters in the initial image, and style features represent the writing style information of the initial image. It should be noted that the conditional encoder includes a text recognition encoder, a text recognition classifier, and a style encoder. The text recognition encoder is configured to generate visual features of the initial image, the text recognition classifier is configured to generate semantic features of the initial image, and the style encoder is configured to generate style features of the initial image.In embodiments of the present disclosure, a pre-trained text recognition model can be directly used as a conditional encoder, or a conditional encoder can be obtained by training multiple training text images with classification labels. In practical applications, there are various ways to input an initial image into the conditional encoder and obtain image features of the initial image. The method of selecting the method depends on the actual situation, and the embodiments of the present disclosure do not impose any restrictions on this method. In one possible implementation of the present disclosure, the initial image can be directly input into the conditional encoder to obtain visual features, semantic features, and style features of the initial image. In another possible implementation of the present disclosure, image features of the initial image can be generated based on at least one generation condition corresponding to the initial image. The image features can be visual features, semantic features, both semantic and visual features, or visual, semantic, and style features. Of course, the image features can also be other combinations of features, and the method of selecting the method depends on the actual situation, and the embodiments of the present disclosure do not impose any restrictions on this method. That is, the aforementioned step of inputting the initial image into the conditional encoder to obtain image features of the initial image may include the following steps: obtaining at least one generation condition corresponding to the initial image, wherein the generation condition is a synthetic generation condition, an augmented generation condition, a restored generation condition, or an imitation generation condition, wherein the image features corresponding to the synthetic generation condition are semantic features, the image features corresponding to the augmented generation condition are visual features, the image features corresponding to the restored generation condition are visual features and semantic features, and the image features corresponding to the imitation generation condition are visual features, semantic features, and style features; and inputting the initial image into the conditional encoder to obtain image features corresponding to the at least one generation condition. Specifically, the synthetic generation condition can be referred to as a synthetic mode. The synthetic generation condition relies solely on semantic features. Text images generated under the synthetic generation condition exhibit rich diversity. For example, handwritten text images may exhibit various character tilts, ink stains, cursive connections, stroke widths, and paper backgrounds, while scene text images may exhibit differences in text rotation, background, blur noise, and fonts. The augmentation generation condition can be called the augmentation mode. It relies solely on visual features. The text images generated under this condition are similar in overall appearance to the original image, but some characters may lose detail. The restoration generation condition can be called the restoration mode. It relies on both semantic and visual features. The text images generated under this condition exhibit both diversity and fidelity. The imitation generation condition can be called the imitation mode. This condition can be used in handwritten text image generation. Because the imitation generation condition adds stylistic characteristics unique to handwritten text images to the restoration generation condition, the fidelity of the text images generated under this condition is further improved, and the corresponding style can be controlled.In practical applications, there are multiple ways to obtain at least one generation condition corresponding to the initial image, and the specific method to be used depends on the actual situation. This disclosure does not impose any restrictions on this method in the embodiments of the present disclosure. In one possible implementation of the present disclosure, the at least one generation condition corresponding to the initial image can be read from another data acquisition device or database. In another possible implementation of the present disclosure, the at least one generation condition corresponding to the initial image can be received from a user. Using the solution of the embodiments of the present disclosure, the at least one generation condition corresponding to the initial image is obtained, the initial image is input into a conditional encoder, and image features corresponding to the at least one generation condition are obtained. By obtaining image features corresponding to actual needs based on the at least one generation condition, unnecessary image features are avoided, the data processing load of the conditional encoder is reduced, and the efficiency of text image generation is improved. In this disclosure, the generation methods of visual features, semantic features, and style features are described in detail. In practical applications, one or more generation methods can be selected from these methods to obtain image features of the initial image. In an optional embodiment of the present disclosure, unlike the patterns in natural images, the rich visual information of text images is concentrated in the text. Therefore, embodiments of the present disclosure propose using a text recognition encoder in a text recognition model to obtain visual features that better represent general information (e.g., texture and color) of a text image rather than noise (e.g., background). Specifically, the conditional encoder includes a text recognition encoder, and the image features include visual features. Inputting the initial image into the conditional encoder to obtain image features corresponding to at least one generation condition may include the following steps: inputting the initial image into the text recognition encoder to obtain initial visual features of the initial image, and based on the initial visual features, obtaining image block indexes of the initial image; performing embedding encoding on the image block indexes to obtain embedded visual features of the initial image; and performing pooling on the initial visual features and embedded visual features to obtain visual features of the initial image. It should be noted that, assuming that the size of the initial visual features obtained after inputting the initial image into the text recognition encoder is H*W*C, where H and W are the height and width of the feature sequence. Since the text recognition encoder processes text lines, H is 1, and C is the number of feature channels after encoding. The image block index of the initial image is Pi, Pi 6 [1, W], which is set to identify the position of the image block in the initial image. The visual features of the initial image are the whole-image features after the image block features are fused. In practical applications, the visual features of the initial image can be calculated by the following formula (1): q = AttnPool(F. cnc⑴ + Emb(Pj)) (1) Wherein, ci is the visual feature, I is the initial image, Fenc is the text recognition encoder, Pi represents the encoded image block index, Emb is the embedding function for embedding Pi to obtain the embedded visual feature, and AttenPool represents the attention pooling operation, which is used to fuse and reduce the dimensionality of sequence features. Applying the solution of the embodiment of the present disclosure, the initial image is input into the text recognition encoder to obtain the initial visual features of the initial image, and based on the initial visual features, the image block index of the initial image is obtained; the image block index is embedded encoded to obtain the embedded visual features of the initial image; the initial visual features and the embedded visual features are pooled to obtain the visual features of the initial image, thereby improving the accuracy of the visual features. In another optional embodiment of the present disclosure, the initial image carries initial text information, the conditional encoder includes a text recognition classifier, and the image features include semantic features. Inputting the initial image into the conditional encoder to obtain image features corresponding to at least one generation condition may include the following steps: inputting the initial text information into the text recognition classifier to obtain semantic features of the initial image, where the semantic features are obtained by processing the initial text sequence by the text recognition classifier, and the initial text sequence is obtained by processing the encoded features of the initial text information by the text recognition classifier. It should be noted that the initial text information refers to the text content in the initial image. Since the input of a typical text recognition model is an image, the image is first converted into features by the encoder and then converted into a text sequence by the classifier. Therefore, in this embodiment of the present disclosure, the text recognition encoder is used to obtain the visual features of the initial image. As for the semantic features of the initial image, the inverse process of the text recognition classifier (input text sequence and output semantic features) can be used to obtain... For example, assuming that the forward process can be understood as the multiplication of the encoding features and the classifier parameters to obtain the initial text sequence. Then the inverse process can be understood as the transposition multiplication of the initial text sequence and the classifier parameters to obtain the semantic features. In practical applications, the semantic features of the initial image can be calculated using the following formula (2): t = Proj(W c T + Emb(P t)) ( 2 ) Wherein, ct is the semantic feature, Proj is the linear projection layer used to normalize the dimension of the semantic feature, Wc is the classifier parameter, T is the one-hot encoding (encoding feature) of each character in the initial text information, Emb is the encoding function for encoding Pt, and Pt represents the index of each character in the initial text information. Assuming that the initial text information is "XY Group", the initial text information is a sequence of 1*W, W=4, then the index of "X" is 1, the index of "Y" is 2, the index of "set" is 3, and the index of "group" is 4. By applying the solution of the embodiment of the present disclosure, since the forward process of generating the initial text sequence and the reverse process of generating the semantic feature use the classifier parameters of the same text recognition classifier, the semantic feature and the encoding feature of the initial text information are more adapted and have a better correlation. Moreover, since the encoding features at the character level of the initial text information are used when generating the semantic feature, a text image containing 00V words can be generated, thereby improving the diversity of text image generation. In another optional embodiment of the present disclosure, the initial image carries initial style information, the conditional encoder includes a style encoder, and the image features include style features; the initial image is input into the conditional encoder to obtain image features corresponding to at least one generation condition, which may include the following steps: inputting the initial style information into the style encoder to obtain style features of the initial image. Specifically, the initial style information does not refer to the overall image style of the initial image, but refers to the writing style information of the writer of the text in the initial image. The initial style information may be a writer number set to represent information such as character tilt, cursive connection, and stroke width. In practical applications, the style features of the initial image can be calculated using the following formula (3): c s= Proj(Emb(S)) (3) Wherein, cs is the style feature, Proj is the linear projection layer used to normalize the dimension of the style feature, S is the initial style information, and Emb is the encoding function for encoding the initial style information. Applying the solution of the embodiment of the present disclosure, the initial style information is input into the style encoder to obtain the style feature of the initial image. In the process of generating the image feature of the initial image, the writer information is introduced to control the style of the handwritten text image, which significantly improves the generation effect of the text image. Step 206: Input the initial image and the image feature into the diffusion generation model to obtain noise data corresponding to the initial image. The diffusion generation model is trained based on the sample text image, the sample image feature of the sample text image, and the noise sample image. The noise sample image is obtained by adding sample noise to the sample text image. The sample image feature includes at least one of a sample visual feature and a sample semantic feature. In one or more embodiments of the present disclosure, after obtaining the initial image and inputting the initial image into the conditional encoder to obtain the image feature of the initial image, the initial image and the image feature can be further input into the diffusion generation model to obtain noise data corresponding to the initial image. Specifically, noise data refers to unnecessary or redundant interference data in the initial image, such as Gaussian noise. Gaussian noise refers to noise whose probability density function follows a Gaussian distribution (i.e., a normal distribution). In practical applications, there are various ways to input the initial image and image features into a diffusion generative model to obtain noise data corresponding to the initial image. The method selected depends on the actual situation and is not limited in this embodiment of the present disclosure. In one possible implementation of the present disclosure, the initial image and image features can be directly input into the diffusion generative model to obtain noise data corresponding to the initial image under all generation conditions, such as synthesized noise data, augmented noise data, restored noise data, and simulated noise data. In another possible implementation of the present disclosure, noise data corresponding to the initial image can be generated based on the initial image and image features corresponding to at least one generation condition. In other words, the aforementioned method of inputting the initial image and image features into the diffusion generative model to obtain noise data corresponding to the initial image can include the following steps: inputting the initial image and image features corresponding to at least one generation condition into the diffusion generative model to obtain noise data corresponding to the initial image under each generation condition. In practical applications, the diffusion generation model can generate noise data corresponding to the initial image under different generation conditions, while the user may only need the target text image under a subset of generation conditions. Therefore, in the embodiments of the present disclosure, a target text image that meets the user's actual needs can be generated based on the generation target of the initial image, that is, at least one generation condition.It should be noted that if the initial image corresponds to multiple generation conditions, the diffusion generative model does not fuse the image features corresponding to different generation conditions when processing the initial image and the image features corresponding to the multiple generation conditions. Instead, it processes the initial image and the image features corresponding to any generation condition separately to obtain noise data corresponding to the initial image under that generation condition. For example, assuming the generation condition corresponding to the initial image is a synthetic generation condition, the initial image and its semantic features can be input into the diffusion generative model to obtain synthetic noise data corresponding to the initial image under the synthetic generation condition. Assuming the generation conditions corresponding to the initial image are both synthetic and augmented generation conditions, the initial image and its visual and semantic features can be input into the diffusion generative model. The diffusion generative model can generate augmented noise data corresponding to the initial image under the augmented generation condition based on the initial image and its visual features, and synthetic noise data corresponding to the initial image under the synthetic generation condition based on the initial image and its semantic features. Using the solution of the embodiments of the present disclosure, an initial image and image features corresponding to at least one generation condition are input into a diffusion generation model to obtain noise data corresponding to the initial image under each generation condition. By obtaining noise data corresponding to the user's actual needs based on the at least one generation condition corresponding to the initial image, unnecessary noise data is avoided, the data processing workload of the diffusion generation model is reduced, and the efficiency of text image generation is improved. Step 208: Generate a target text image corresponding to the initial image based on the initial image and the noise data. In one or more embodiments of the present disclosure, after obtaining an initial image, inputting the initial image into a conditional encoder to obtain image features of the initial image, and inputting the initial image and image features into a diffusion generation model to obtain noise data corresponding to the initial image, a target text image corresponding to the initial image can be generated based on the initial image and the noise data. Specifically, the target text image corresponding to the initial image can be of one or more types, depending on actual circumstances, and is not limited in this embodiment. The target text image corresponds to the generation condition corresponding to the initial image. Types of target text images include, but are not limited to, synthesized text images, augmented text images, restored text images, and simulated text images. Furthermore, the number of target text images of a type can be one or more, depending on the user's actual situation. This disclosure does not impose any limitations on this. For example, assuming that the generation conditions corresponding to the initial image are synthetic generation conditions, the target text images corresponding to the initial image generated according to the text image generation method can be multiple synthetic text images.Assuming that the generation conditions corresponding to the initial image are synthetic generation conditions and augmented generation conditions, the target text image corresponding to the initial image generated according to the text image generation method may include a synthetic text image and multiple augmented text images. In practical applications, when generating the target text image corresponding to the initial image based on the initial image and noise data, the initial image and noise data can be subtracted to generate the target text image corresponding to the initial image. By applying the solution of the embodiments of the present disclosure, since sample noise is introduced when training the diffusion generation model, an infinite number of noise sample images can be generated based on the sample noise, thus expanding the training data of the diffusion generation model. In addition, sample image features related to the sample text images are introduced when training the diffusion generation model, effectively encoding the contextual information and texture features between the characters in the sample text images, significantly improving the quality of the text images generated by the diffusion generation model. In the embodiments of the present disclosure, during the text image generation process, multiple target text images can be generated and sent to the user. Furthermore, a text image that meets the user's requirements can be screened from the multiple target text images and sent to the user. In an optional embodiment of the present disclosure, a target text image includes a first target text image and a second target text image. Generating a target text image corresponding to the initial image based on the initial image and noise data may include the following steps: generating a first target text image corresponding to the initial image based on the initial image and noise data; using the first target text image as the updated initial image and returning to the step of inputting the initial image into a conditional encoder to obtain image features of the initial image, thereby generating a second target text image corresponding to the initial image. It should be noted that the first target text image and the second target text image may be of the same or different types, depending on actual circumstances, and this embodiment of the disclosure does not impose any specific restrictions on this. For example, assume that the initial image sent by the user is initial image A, and that two text images corresponding to the initial image are generated, with the generation condition corresponding to initial image A being a synthetic generation condition. Initial image A is input into the conditional encoder to obtain semantic features of the initial image under the synthetic generation condition. The initial image and the semantic features are input into a diffusion generation model to obtain synthetic noise data corresponding to the initial image. A first synthetic text image corresponding to the initial image is generated based on the initial image and the synthetic noise data. Since only one text image has been generated, the first synthetic text image can be input into the conditional encoder to obtain semantic features of the first synthetic text image under the synthetic generation conditions. The first synthetic text image and its semantic features are then input into the diffusion generation model to obtain synthetic noise data corresponding to the first synthetic text image.A second synthetic text image corresponding to the first synthetic text image is generated based on the first synthetic text image and synthetic noise data of the first synthetic text image. At this point, two text images, the first synthetic text image and the second synthetic text image, are generated. Text image generation is stopped, and the first synthetic text image and the second synthetic text image are sent to the user. The clarity of the second synthetic text image can be greater than that of the first synthetic text image. Applying the solution of an embodiment of the present disclosure, a first target text image corresponding to the initial image is generated based on the initial image and noise data. The first target text image is used as the updated initial image, and the process returns to the step of inputting the initial image into a conditional encoder to obtain image features of the initial image, thereby generating a second target text image corresponding to the initial image. Generating target text images corresponding to the initial image in an iterative manner increases the diversity of the generated target text images. In an optional embodiment of the present disclosure, after generating the target text image corresponding to the initial image based on the initial image and noise data, the target text image can be directly sent to the user. Alternatively, the process can receive text image generation requirement information sent by the user and determine whether the currently generated target image meets the text image generation requirement information. If the requirements are not met, the currently generated target image is used as the initial image to regenerate the target text image. If the requirements are met, the target text image is directly sent to the user. Specifically, after generating the target text image corresponding to the initial image based on the initial image and noise data, the method may further include the following steps: receiving text image generation requirement information from the user; if the target text image does not meet the image generation requirement information, using the target text image as the updated initial image and returning to the step of inputting the initial image into the conditional encoder to obtain image features of the initial image, until the target text image meets the image generation requirement information, thereby obtaining an updated target text image. Specifically, the text image generation requirement information includes, but is not limited to, the number of text images and the clarity of the text images, which are selected based on actual circumstances and are not limited in any way in the presently disclosed embodiments. For example, after generating the target text image corresponding to the initial image based on the initial image and noise data, the method may receive text image generation requirement information from the user, such as whether the clarity meets a preset clarity threshold. The clarity of the target text image is compared with a preset clarity threshold, and it is determined that the clarity of the target text image is less than the preset clarity threshold, that is, the target text image does not meet the image generation requirement information.At this point, the target text image can be used as the updated initial image, and the process returns to the step of inputting the initial image into the conditional encoder to obtain image features of the initial image. This results in an updated target text image. The clarity of the updated target text image is then compared with a preset clarity threshold. If the clarity of the updated target text image exceeds the preset clarity threshold, it is determined that the updated target text image meets the image generation requirement information. The updated target text image is then sent to the user. Using the solution of the disclosed embodiment, the text image generation requirement information sent by the user is received. Based on the text image generation requirement information, a determination is made as to whether the currently generated target text image should be sent to the user. If not, a target text image that meets the text image generation requirement information is regenerated and sent to the user. This ensures the accuracy of text image generation and further improves the user experience. Optical character recognition technology is one of the most successful applications of pattern recognition and has extremely high research value. Text recognition is a key step in optical character recognition. Because it's difficult to achieve the complexity and diversity found in the real world by collecting and annotating limited real-world text image data, training text recognition models requires expanding training samples, such as through data synthesis and data augmentation. Currently, training sample expansion can be achieved through the following methods: Real-world data collection and annotation: Collecting and annotating real-world data is extremely time-consuming and labor-intensive. Sample diversity is often limited by the collected fonts, backgrounds, and manually designed rules, and it's impossible to generate high-quality, unconstrained handwritten text images. Data augmentation: Data augmentation involves geometric and feature-level augmentation of the original image. However, data augmentation is typically based on existing training data and cannot create out-of-the-box (OOV) data, thus limiting sample diversity. Data synthesis: Generate sample text images using a generative adversarial network. However, this method is often difficult to train and prone to mode collapse without carefully selected hyperparameters and regularizers. In practical applications, natural-looking images can be generated by training a diffusion-based generative model and then gradually denoising it with random Gaussian noise during inference. However, without prior information, the diffusion generative model produces highly random output images, making them unusable. Therefore, a pre-trained text image model can be used to align images with their corresponding text descriptions, which can then be used as additional input to the diffusion generative model. This allows the generated text images to have a certain degree of meaning. Furthermore, unlike general natural image generation, text image generation requires more unique contextual information and texture features between characters.Therefore, in the disclosed embodiments, the text image generation method provided in FIG. 2 can be used to expand the training data of the text recognition model, thereby obtaining a target text recognition model suitable for text images. Specifically, after generating a target text image corresponding to the initial image based on the initial image and noise data, the following steps may be further included: constructing a text recognition training set based on multiple initial images and the target text images corresponding to each initial image; and training the text recognition model using the text recognition training set to obtain the target text recognition model. It should be noted that since the target text image is obtained based on the initial image and the noise data corresponding to the initial image, the text labels corresponding to the target text image and the initial image are the same. Therefore, both the initial image and the target text image corresponding to the initial image can be used as text recognition samples, and the text recognition training set can be constructed using the multiple text recognition samples and the text labels of each text recognition sample. In practical applications, training a target text recognition model may include the following steps: obtaining a text recognition training set, wherein the text recognition training set includes multiple text recognition samples, each of which carries text labels; extracting a first text recognition sample from the text recognition training set, wherein the first text recognition sample is any one of the multiple text recognition samples; inputting the first text recognition sample into the text recognition model to obtain a first predicted text for the first text recognition sample; calculating a recognition loss value based on the first predicted text and the first text label carried by the first text recognition sample; adjusting parameters of the text recognition model based on the recognition loss value, and returning to the step of extracting the first text recognition sample from the text recognition training set until a preset recognition stop condition is met, thereby obtaining a trained target text recognition model. Using the solution of the embodiments of the present disclosure, a text recognition training set is constructed based on multiple initial images and target text images corresponding to each initial image; and a text recognition model is trained using the text recognition training set to obtain a target text recognition model. By combining three image features (visual, semantic, and style) and four generation conditions (synthetic, augmented, restored, and imitative), the generated target text images exhibit superior effectiveness, fidelity, and diversity. Furthermore, the generated target text images can significantly improve the performance of text recognition models and complement other data augmentation schemes to address text recognition problems in real-world text images. Furthermore, because the synthetic generation condition is independent of visual features, this scheme also demonstrates superior performance in domain transfer and out-of-the-box (OOV) text image generation. It is simple and stable to use, capable of generating unlimited images across the entire distribution. Furthermore, the inference process is highly flexible, allowing control over the diversity and fidelity of the generated images, as well as the handwriting style of handwritten text images.In an optional embodiment of the present disclosure, a conditional encoder includes a text recognition encoder and a text recognition classifier. Training the conditional encoder may include the following steps: obtaining a training set, wherein the training set includes multiple training text images, each of which carries a classification label; extracting a first training text image from the training set, wherein the first training text image is any one of the multiple training text images; inputting the first training text image into the text recognition encoder, and using the output of the text recognition encoder as input to the text recognition classifier to obtain a first predicted classification result for the first training text image; calculating an encoding loss value based on the first predicted classification result and the first classification label carried by the first training text image; adjusting parameters of the text recognition encoder and the text recognition classifier based on the encoding loss value, and returning to the step of extracting the first training text image from the training set until a first preset stop condition is met, thereby obtaining a trained conditional encoder. It should be noted that the training method of the conditional encoder is supervised training, that is, each training text image in the training set carries a true classification label. The classification label is a processing target of the conditional encoder and is set to guide the training process of the conditional encoder. The training set can be obtained by reading a large number of training text images carrying classification labels from other data acquisition devices or databases to form a training set. Alternatively, a training set may be formed by receiving a large number of training text images with classification labels input by the user. The method for obtaining the training set is selected based on actual circumstances and is not limited in any way in the embodiments of the present disclosure. In one possible implementation of the present disclosure, the first preset stopping condition includes a coding loss value being less than or equal to a first preset threshold. After calculating the coding loss value based on the first predicted classification result and the first classification label carried by the first training text image, the coding loss value is compared with the first preset threshold. Specifically, if the coding loss value is greater than the first preset threshold, it indicates that the difference between the first predicted classification result and the first classification label is large, and the conditional encoder has poor prediction ability for the first predicted classification result. In this case, the parameters of the text recognition encoder and the text recognition classifier can be adjusted, and the process returns to the step of extracting the first training text image from the training set. The conditional encoder is trained continuously until the coding loss value is less than or equal to the first preset threshold. This indicates that the difference between the first predicted classification result and the first classification label is small, and the conditional encoder has strong prediction ability. The first preset stopping condition is met, and training can be stopped to obtain a trained conditional encoder. In another possible implementation of the present disclosure, in addition to comparing the coding loss value with the first preset threshold, It is also possible to determine whether the current conditional encoder has been trained in combination with the number of first iterations.Specifically, if the encoding loss value is greater than a first preset threshold, the parameters of the text recognition encoder and the text recognition classifier are adjusted, and the process returns to the step of extracting the first training text image from the training set. The conditional encoder continues training until a first preset number of iterations is reached, at which point the iterations are stopped, resulting in a fully trained conditional encoder. The first preset threshold and the first preset number of iterations are selected based on practical circumstances and are not limited in this embodiment. In practical applications, many functions are used to calculate the encoding loss value, such as the cross entropy loss function, the L1 norm loss function, the maximum loss function, the mean square error loss function, the logarithmic loss function, etc. These functions are selected based on practical circumstances and are not limited in this embodiment. Using the solution of this embodiment, an encoding loss value is calculated based on the first predicted classification result and the first classification label carried by the first training text image. The encoding loss value is then compared with a first preset stopping condition. If the first preset stopping condition is not met, training of the text recognition encoder and the text recognition classifier continues until the first preset stopping condition is reached, completing training and obtaining the conditional encoder. By continuously adjusting the parameters of the text recognition encoder and the text recognition classifier, the resulting conditional encoder can be made more accurate. In an optional embodiment of the present disclosure, a method for training a diffusion generative model may include the following steps: obtaining a sample set, wherein the sample set includes multiple sample text images, each of which carries sample image features; extracting a first sample text image from the sample set, wherein the first sample text image is any one of the multiple sample text images; adding a first sample noise to the first sample text image to obtain a first noisy sample image; inputting the first noisy sample image and the first sample image features carried by the first sample text image into the diffusion generative model to obtain a first predicted noise; calculating a noise loss value based on the first sample noise and the first predicted noise; adjusting the model parameters of the diffusion generative model based on the noise loss value, and returning to the step of extracting the first sample text image from the sample set until a second preset stop condition is met, thereby obtaining a trained diffusion generative model. Specifically, the sample image features carried by the sample text image are obtained by processing the sample text image using the conditional encoder. The sample image features include at least one of sample visual features and sample semantic features, and may also include sample style features. The first sample noise can be random noise or fixed noise, depending on the actual situation. This disclosure does not impose any limitation on this aspect. In practical applications, the sample set can be obtained by reading a large number of sample text images carrying sample image features from other data acquisition devices or databases to form the sample set.Alternatively, a sample set may be formed by receiving a large number of sample text images containing sample image features input by the user. The method for obtaining the sample set is selected based on actual circumstances. In one possible implementation of the present disclosure, the second preset stopping condition includes a noise loss value being less than or equal to a second preset threshold. After calculating the noise loss value based on the first sample noise and the first predicted noise, the noise loss value is compared with the second preset threshold. Specifically, if the noise loss value is greater than the second preset threshold, it indicates that the difference between the first sample noise and the first predicted noise is large, and the prediction capability of the diffusion generation model for the first predicted noise is poor. In this case, the parameters of the diffusion generation model can be adjusted, and the process returns to the step of extracting the first sample text image from the sample set. The diffusion generation model is then trained again until the noise loss value is less than or equal to the second preset threshold. This indicates that the difference between the first sample noise and the first predicted noise is small, and the prediction capability of the diffusion generation model is strong. The second preset stopping condition is met, and training can be stopped to obtain a trained diffusion generation model. In another possible implementation of the present disclosure, in addition to comparing the noise loss value with the second preset threshold, the second number of iterations can also be used to determine whether the current diffusion generation model is trained. Specifically, if the noise loss value is greater than a second preset threshold, the parameters of the diffusion generative model are adjusted, and the process returns to the step of extracting the first sample text image from the sample set. The diffusion generative model is trained continuously until a second preset number of iterations is reached, at which point the iterations are stopped, resulting in a fully trained diffusion generative model. The second preset threshold and the second preset number of iterations are selected based on practical circumstances and are not limited in this embodiment. In practical applications, many functions are used to calculate noise loss values, such as the cross entropy loss function, the L1 norm loss function, the maximum loss function, the squared error loss function, the logarithmic loss function, and so on. These functions are selected based on practical circumstances. Using the solution of this embodiment, a noise loss value is calculated based on the first sample noise and the first predicted noise. The noise loss value is then compared with a second preset stopping condition. If the second preset stopping condition is not met, training of the diffusion generative model is continued until the second preset stopping condition is reached, completing the training and obtaining the diffusion generative model. By continuously adjusting the parameters of the diffusion generative model, the resulting diffusion generative model can be made more accurate. 3 , which shows a flowchart of a processing process of a diffusion model training method provided by an embodiment of the present disclosure.The conditional diffusion model training process uses a conditional encoder and a diffusion generative model: The conditional encoder inputs a sample text image into a text recognition encoder to obtain initial visual features of the sample text image. Based on the initial visual features, image block indexes of the sample text image are obtained. Embedding encoding is performed on the image block indexes to obtain embedded visual features of the sample text image. Pooling is performed on the initial and embedded visual features to obtain sample visual features of the sample text image. Sample text information of the sample text image is input into a text recognition classifier. Linear projection is performed on the output of the text recognition classifier and the character indexes of the sample text information to obtain sample semantic features of the sample text image. Sample style information of the sample text image is input into a style encoder. Linear projection is performed on the output of the style encoder to obtain sample style features of the sample text image. The diffusion generative model: Unlike the natural image generation in conventional diffusion models, this embodiment incorporates expert knowledge related to the sample text image, namely, sample image features including sample visual features, sample semantic features, and sample style features. In practical applications, the diffusion generation model can be trained based on an encoder-decoder network (such as UNet) that can make the input and output of the same scale. By continuously adding random Gaussian noise to the sample text image, the Where L represents the squared error loss value, g represents Gaussian noise, wind represents the parameters of the UNet network, [ci, ct, cs] represents the cascade operation of visual features, semantic features, and style features, and In+1 represents the noise sample image of step n+1. Referring to Figure 4, Figure 4 shows a flowchart of the processing process of a text image generation method provided by one embodiment of the present disclosure. The text image generation process uses a conditional encoder and a diffusion generation model: The conditional encoder: Inputs the initial image into the text recognition encoder to obtain initial visual features of the initial image, and based on the initial visual features, obtains image block indexes of the initial image; performs embedding encoding on the image block indexes to obtain embedded visual features of the initial image; and performs pooling on the initial visual features and embedded visual features to obtain visual features of the initial image. Inputs the initial text information of the initial image into the text recognition classifier, linearly projects the output of the text recognition classifier and the character indexes of the initial text information to obtain semantic features of the initial image. Inputs the initial style information of the initial image into the style encoder, and linearly projects the output of the style encoder to obtain style features of the initial image. Diffusion Generation Model: The target text image is generated by gradually denoising the initial image using random Gaussian noise. Specifically, in the nth step of the generation process, different generation conditions and the denoised text image In are combined and fed into the UNet to predict the noise of the denoised text image. The combination process uses a cross-attention mechanism to promote information interaction and learning. Therefore, the next denoised text image In+1 can be obtained by subtracting In from the predicted noise. In practical applications, the denoised text image can be determined using the following formula (6):
[0004] 1瞄 = 1 factory and 0 (6) where, In,+1 represents the denoised text image at the (n'+1)-th step, In represents the denoised text image at the n'-th step, c represents different combined settings of visual features, semantic features, and style features, that is, the image features corresponding to different generation conditions, and % represents the parameters of the UNet network. The text image generation method provided by one or more embodiments of the present disclosure can be applied to different text image generation scenarios, such as natural scene text recognition covering various street scene backgrounds and fonts, and handwritten scenarios with individualized writing styles. It can also be applied to various types or language text image generation fields such as Chinese text generation, ancient book text generation, and typo generation. And, since the text image generation method can generate images in a specified author's style, it can also be applied to the field of signature authentication or author recognition, that is, classifying the corresponding author based on the image. The following combines FIG. 5 and takes the application of the text image generation method provided by the present disclosure in the handwritten scenario as an example to further illustrate the text image generation method. Among them, FIG. 5 shows a flowchart of a handwritten text image generation method provided by an embodiment of the present disclosure, which specifically includes the following steps: Step 502: Receive a handwritten text image generation request sent by the user, where the handwritten text image generation request carries an initial image and initial style information of the initial image. Step 504: Input the initial image and initial style information into the conditional encoder to obtain the visual features, semantic features, and style features of the initial image. Step 506: Input the initial image, visual features, semantic features, and style features into the diffusion generation model to obtain the noise data corresponding to the initial image, where the diffusion generation model is trained based on sample text images, sample image features of the sample text images, and noise sample images, and the noise sample images are obtained by adding sample noise to the sample text images, and the sample image features include sample visual features, sample semantic features, and sample style features. Step 508: Generate the target handwritten text image corresponding to the initial image according to the initial image and the noise data. Step 510: Send the target handwritten text image corresponding to the initial image to the user. It should be noted that the initial image can be an initial handwritten text image including handwritten text content, or an initial noise image carrying initial text information, where the initial noise image does not include text content. The specific implementation manners of steps 502 to 508 are the same as those of steps 202 to 208 above, and the embodiments of the present disclosure will not be elaborated herein.The solution of the embodiments of the present disclosure introduces sample noise when training the diffusion generative model. This allows for the generation of an infinite number of noise sample images based on the sample noise, expanding the training data for the diffusion generative model and making the training and inference process extremely simple and stable. Furthermore, since the generation target of this solution is text images, which prioritize the details of text rather than its overall appearance, for example, generating meaningful characters rather than meaningless symbols, this solution incorporates three types of prior knowledge highly relevant to text images: visual features, semantic features, and style features. This effectively encodes the unique contextual information and texture features between characters in text images, enhancing the representation of text images and ensuring the generated text images possess outstanding effectiveness, fidelity, and diversity. Furthermore, by introducing style information into the handwritten text image generation process to control the style of the handwritten text image, a diverse range of handwritten text images can be generated without the need to collect a variety of fonts and background materials or manually design synthesis rules, significantly improving the quality of the generated target handwritten text images. See Figure 6, which shows a schematic diagram of a text image generated in a handwriting scenario, according to one embodiment of the present disclosure. Using the text image generation method provided in FIG2 , the handwritten texts "accept," "concert," "Macmillan," "conference," "treatments," and "waxed" and their corresponding initial images are processed to generate two target text images corresponding to each initial image under synthesis generation conditions, augmentation generation conditions, restoration generation conditions, and imitation generation conditions, as shown in FIG6 . Referring to FIG7 , a schematic diagram of a text image generated in a natural scene according to an embodiment of the present disclosure is shown. Using the text image generation method provided in FIG2 , text 1 in a natural scene is processed. 1ACHIEVED, "Cellular," "jacket," "easylake," "No entry," and "Share," along with the initial images corresponding to each text, can be processed to generate three target text images corresponding to each initial image under the synthesis generation condition, augmentation generation condition, and restoration generation condition, as shown in FIG7 . Referring to FIG8 , FIG8 shows a flowchart of a diffusion generation model training method provided by one embodiment of the present disclosure. This diffusion generation model training method is applied to a cloud instance device and specifically includes the following steps: Step 802: Acquire a sample set, wherein the sample set includes multiple sample text images, each of which carries sample image features, including at least one of sample visual features and sample semantic features. Step 804: Extract a first sample text image from the sample set, wherein the first sample text image is any one of the multiple sample text images. Step 806: Add a first sample noise to the first sample text image to obtain a first noise sample image. Step 808: Input the first noise sample image and the first sample image features carried by the first sample text image into the diffusion generation model to obtain a first predicted noise. Step 810: Calculate a noise loss value based on the first sample noise and the first predicted noise. Step 812: Adjust the model parameters of the diffusion generation model based on the noise loss value, and return to the step of extracting the first sample text image from the sample set until a second preset stop condition is met, thereby obtaining the model parameters of the trained diffusion generation model. Step 814: Send the model parameters of the diffusion generation model to the end-side device. It should be noted that steps 802 to 814 are implemented in the same manner as steps 202 to 208 above, and will not be further described in this embodiment of the present disclosure. In actual applications, after the cloud-side device sends the model parameters of the diffusion generation model to the end-side device, the end-side device can locally construct the diffusion generation model based on the model parameters of the diffusion generation model and further use the diffusion generation model to generate text images. Using the solution of this embodiment of the present disclosure, a noise loss value is calculated based on the first sample noise and the first predicted noise. The noise loss value is compared with the second preset stop condition. If the second preset stop condition is not met, training of the diffusion generation model continues until the second preset stop condition is met, completing the training and obtaining the diffusion generation model. By continuously adjusting the parameters of the diffusion generation model, the resulting diffusion generation model can be made more accurate. Corresponding to the above-described method embodiments, the present disclosure also provides embodiments of a text image generation device. FIG9 shows a schematic structural diagram of a text image generation device provided by one embodiment of the present disclosure.As shown in Figure 9, the apparatus includes: a first acquisition module 902, configured to acquire an initial image; a first input module 904, configured to input the initial image into a conditional encoder to obtain image features of the initial image; a second input module 906, configured to input the initial image and the image features into a diffusion generation model to obtain noise data corresponding to the initial image, wherein the diffusion generation model is trained based on a sample text image, sample image features of the sample text image, and a noise sample image, the noise sample image is obtained by adding sample noise to the sample text image, and the sample image features include at least one of sample visual features and sample semantic features; and a first generation module 908, configured to generate a target text image corresponding to the initial image based on the initial image and the noise data. Optionally, the first input module 904 is further configured to obtain at least one generation condition corresponding to the initial image, wherein the generation condition is a synthesis generation condition, an augmentation generation condition, a restoration generation condition, or an imitation generation condition, the image features corresponding to the synthesis generation condition are semantic features, the image features corresponding to the augmentation generation condition are visual features, the image features corresponding to the restoration generation condition are visual features and semantic features, and the image features corresponding to the imitation generation condition are visual features, semantic features, and style features; input the initial image into a conditional encoder to obtain image features corresponding to the at least one generation condition. Optionally, the conditional encoder includes a text recognition encoder, and the image features include visual features; the first input module 904 is further configured to input the initial image into the text recognition encoder to obtain initial visual features of the initial image, and based on the initial visual features, obtain image block indexes of the initial image; perform embedding coding on the image block indexes to obtain embedded visual features of the initial image; and perform pooling on the initial visual features and the embedded visual features to obtain visual features of the initial image. Optionally, the initial image carries initial text information, the conditional encoder includes a text recognition classifier, and the image features include semantic features. The first input module 904 is further configured to input the initial text information into the text recognition classifier to obtain semantic features of the initial image, where the semantic features are obtained by processing the initial text sequence by the text recognition classifier, and the initial text sequence is obtained by processing the encoding features of the initial text information by the text recognition classifier. Optionally, the initial image carries initial style information, the conditional encoder includes a style encoder, and the image features include style features. The first input module 904 is further configured to input the initial style information into the style encoder to obtain style features of the initial image.Optionally, the second input module 906 is further configured to input the initial image and image features corresponding to at least one generation condition into the diffusion generation model to obtain noise data corresponding to the initial image under each generation condition. Optionally, the target text image includes a first target text image and a second target text image; the first generation module 908 is further configured to generate a first target text image corresponding to the initial image based on the initial image and the noise data; use the first target text image as the updated initial image, and return to the step of inputting the initial image into the conditional encoder to obtain image features of the initial image, thereby generating a second target text image corresponding to the initial image. Optionally, the apparatus further includes: a text image generation requirement information receiving module configured to receive text image generation requirement information sent by a user; if the target text image does not meet the image generation requirement information, use the target text image as the updated initial image, and return to the step of inputting the initial image into the conditional encoder to obtain image features of the initial image, until the target text image meets the image generation requirement information, thereby obtaining an updated target text image. Optionally, the conditional encoder includes a text recognition encoder and a text recognition classifier, and the device further includes: a conditional encoder training module, configured to obtain a training set, wherein the training set includes multiple training text images, and the training text images carry classification labels; extract a first training text image from the training set, wherein the first training text image is any one of the multiple training text images; input the first training text image into the text recognition encoder, and use the output of the text recognition encoder as the input of the text recognition classifier to obtain a first predicted classification result of the first training text image; calculate an encoding loss value based on the first predicted classification result and the first classification label carried by the first training text image; adjust parameters of the text recognition encoder and the text recognition classifier based on the encoding loss value, and return to the step of extracting the first training text image from the training set until a first preset stop condition is met, thereby obtaining a trained conditional encoder.Optionally, the apparatus further includes: a diffusion generation model training module configured to obtain a sample set, wherein the sample set includes multiple sample text images, and the sample text images carry sample image features; extract a first sample text image from the sample set, wherein the first sample text image is any one of the multiple sample text images; add a first sample noise to the first sample text image to obtain a first noise sample image; input the first noise sample image and the first sample image features carried by the first sample text image into the diffusion generation model to obtain a first predicted noise; calculate a noise loss value based on the first sample noise and the first predicted noise; adjust model parameters of the diffusion generation model based on the noise loss value, and return to the step of extracting the first sample text image from the sample set until a second preset stop condition is met to obtain a trained diffusion generation model. According to the solution of the embodiment of the present disclosure, since sample noise is introduced when training the diffusion generation model, an infinite number of noise sample images can be generated based on the sample noise, thereby expanding the training data of the diffusion generation model. In addition, sample image features related to the sample text image are also introduced when training the diffusion generation model, effectively encoding contextual information and texture features between characters in the sample text image. The quality of text images generated by the diffusion generation model is significantly improved. The above is a schematic diagram of a text image generation device according to this embodiment. It should be noted that the technical solution of this text image generation device shares the same concept as the technical solution of the aforementioned text image generation method. For details not described in detail in the technical solution of the text image generation device, please refer to the description of the technical solution of the aforementioned text image generation method. Corresponding to the aforementioned method embodiment, the present disclosure also provides an embodiment of a handwritten text image generation device. Figure 10 shows a schematic diagram of the structure of a handwritten text image generation device according to one embodiment of the present disclosure.As shown in FIG10 , the apparatus includes: a receiving module 1002 configured to receive a handwritten text image generation request sent by a user, wherein the handwritten text image generation request carries an initial image and initial style information of the initial image; a third input module 1004 configured to input the initial image and the initial style information into a conditional encoder to obtain visual features, semantic features, and style features of the initial image; a fourth input module 1006 configured to input the initial image, the visual features, the semantic features, and the style features into a diffusion generation model to obtain noise data corresponding to the initial image, wherein the diffusion generation model is trained based on a sample text image, sample image features of the sample text image, and a noise sample image, wherein the noise sample image is obtained by adding sample noise to the sample text image, and the sample image features include sample visual features, sample semantic features, and sample style features; a second generation module 1008 configured to generate a target handwritten text image corresponding to the initial image based on the initial image and the noise data; and a first sending module 1010 configured to send the target handwritten text image corresponding to the initial image to the user. The solution of the disclosed embodiments introduces sample noise when training the diffusion generative model. This allows for the generation of an infinite number of noise sample images based on the sample noise, expanding the training data for the diffusion generative model and making the training and inference process significantly simpler and more stable. Furthermore, since the generation target of this solution is text images, which prioritize the details of text rather than its overall appearance—for example, generating meaningful characters rather than meaningless symbols—this solution incorporates three types of prior knowledge highly relevant to text images: visual features, semantic features, and style features. This effectively encodes the unique contextual information and texture features between characters in text images, enhancing the representation of text images and ensuring the generated text images possess outstanding effectiveness, fidelity, and diversity. Furthermore, by incorporating style information into the handwritten text image generation process to control the style of the handwritten text image, a diverse range of handwritten text images can be generated without the need to collect a wide variety of fonts and background materials or manually design synthesis rules, significantly improving the quality of the generated target handwritten text images. The above is a schematic diagram of a handwritten text image generation device according to this embodiment. It should be noted that the technical solutions of this handwritten text image generation device and the aforementioned handwritten text image generation method share the same concept. For details not described in detail in the technical solution of the handwritten text image generation device, please refer to the description of the technical solution of the aforementioned handwritten text image generation method. Corresponding to the aforementioned method embodiments, the present disclosure also provides an embodiment of a diffusion generation model training device. Figure 11 shows a schematic structural diagram of a diffusion generation model training device provided in one embodiment of the present disclosure.As shown in FIG11 , the apparatus is applied to a cloud-side device and includes: a second acquisition module 1102 configured to acquire a sample set, wherein the sample set includes multiple sample text images, the sample text images carry sample image features, and the sample image features include at least one of sample visual features and sample semantic features; an extraction module 1104 configured to extract a first sample text image from the sample set, wherein the first sample text image is any one of the multiple sample text images; an addition module 1106 configured to add a first sample noise to the first sample text image to obtain a first noise sample image; a fifth input module 1108 configured to input the first noise sample image and the first sample image features carried by the first sample text image into a diffusion generation model to obtain a first predicted noise; a calculation module 1110 configured to calculate a noise loss value based on the first sample noise and the first predicted noise; and an adjustment module 1112 configured to adjust model parameters of the diffusion generation model based on the noise loss value and return to the step of extracting the first sample text image from the sample set. Until a second preset stop condition is met, the model parameters of the trained diffusion generation model are obtained. The second sending module 1114 is configured to send the model parameters of the diffusion generation model to the terminal device. Applying the solution of an embodiment of the present disclosure, a noise loss value is calculated based on the first sample noise and the first predicted noise. The noise loss value is compared with the second preset stop condition. If the second preset stop condition is not met, the diffusion generation model is trained continuously until the second preset stop condition is met, completing the training and obtaining the diffusion generation model. By continuously adjusting the parameters of the diffusion generation model, the resulting diffusion generation model can be made more accurate. The above is a schematic diagram of a diffusion generation model training device according to this embodiment. It should be noted that the technical solution of the diffusion generation model training device and the technical solution of the diffusion generation model training method described above are based on the same concept. For details not described in detail in the technical solution of the diffusion generation model training device, please refer to the description of the technical solution of the diffusion generation model training method described above. Figure 12 shows a block diagram of a computing device provided in one embodiment of the present disclosure. Components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230. The database 1250 is configured to store data. The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260.Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. Access device 1240 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a World Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like. In one embodiment of the present disclosure, the aforementioned components of computing device 1200 and other components not shown in FIG. 12 may also be connected to each other, for example, via a bus. It should be understood that the computing device block diagram shown in FIG12 is for illustrative purposes only and does not limit the scope of this disclosure. Those skilled in the art may add or replace other components as needed. Computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1200 may also be a mobile or stationary server. Processor 1220 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the aforementioned text image generation method, handwritten text image generation method, or diffusion generative model training method. The above is a schematic diagram of a computing device in this embodiment.It should be noted that the technical solution of this computing device is based on the same concept as the technical solutions of the aforementioned text image generation method, handwritten text image generation method, and diffusion generation model training method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the aforementioned text image generation method, handwritten text image generation method, or diffusion generation model training method. An embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the aforementioned text image generation method, handwritten text image generation method, or diffusion generation model training method. The above is a schematic diagram of a computer-readable storage medium of this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solutions of the aforementioned text image generation method, handwritten text image generation method, and diffusion generation model training method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the aforementioned text image generation method, handwritten text image generation method, or diffusion generation model training method. One embodiment of the present disclosure further provides a computer program. When executed on a computer, the computer program causes the computer to perform the steps of the aforementioned text image generation method, handwritten text image generation method, or diffusion generative model training method. The above is an illustrative embodiment of a computer program according to this embodiment. It should be noted that the technical solution of this computer program shares the same concept as the technical solutions of the aforementioned text image generation method, handwritten text image generation method, and diffusion generative model training method. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the aforementioned text image generation method, handwritten text image generation method, or diffusion generative model training method. The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous. The computer instructions include computer program codes, which may be in source code form, object code form, executable file, or some intermediate form.The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. It should be noted that, for ease of description, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present disclosure are not limited by the order of the actions described, as, depending on the embodiments of the present disclosure, certain steps may be performed in a different order or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this disclosure are preferred embodiments, and the actions and modules involved are not necessarily required for the embodiments of the present disclosure. In the above embodiments, the description of each embodiment has its own emphasis. For portions not described in detail in a particular embodiment, reference should be made to the relevant descriptions of other embodiments. The preferred embodiments of the present disclosure disclosed above are merely intended to help illustrate the present disclosure. The optional embodiments do not describe all details in detail, nor do they limit the invention to the specific implementations described. Obviously, many modifications and variations are possible based on the content of the embodiments of this disclosure. This disclosure selects and specifically describes these embodiments to better explain the principles and practical applications of the embodiments of this disclosure, thereby enabling those skilled in the art to better understand and utilize the disclosure. The disclosure is limited by the claims and their full scope and equivalents. Industrial Applicability The text image generation method provided in the embodiments of this disclosure comprises obtaining an initial image; inputting the initial image into a conditional encoder to obtain image features of the initial image; inputting the initial image and the image features into a diffusion generation model to obtain noise data corresponding to the initial image. The diffusion generation model is trained based on a sample text image, sample image features of the sample text image, and a noise sample image. The noise sample image is obtained by adding sample noise to the sample text image. The sample image features include at least one of sample visual features and sample semantic features. Based on the initial image and the noise data, a target text image corresponding to the initial image is generated. Since sample noise is introduced when training the diffusion generative model, an infinite number of noise sample images can be generated based on the sample noise, which expands the training data of the diffusion generative model. In addition, sample image features related to sample text images are also introduced when training the diffusion generative model, effectively encoding the contextual information and texture features between characters in the sample text images, significantly improving the quality of the text images generated by the diffusion generative model.
Claims
Claims 1. A text image generation method, comprising: Acquiring an initial image; Inputting the initial image into a conditional encoder to obtain image features of the initial image; The initial image and the image features are input into a diffusion generation model to obtain noise data corresponding to the initial image, wherein the diffusion generation model is trained based on a sample text image, sample image features of the sample text image and a noise sample image, the noise sample image is obtained by adding sample noise to the sample text image, and the sample image features include at least one of sample visual features and sample semantic features; and a target text image corresponding to the initial image is generated according to the initial image and the noise data.
2. The method according to claim 1, wherein inputting the initial image into a conditional encoder to obtain image features of the initial image comprises: Acquire at least one generation condition corresponding to the initial image, wherein the generation condition is a synthesis generation condition or an augmentation generation condition or a restoration generation condition or an imitation generation condition, the image feature corresponding to the synthesis generation condition is a semantic feature, the image feature corresponding to the augmentation generation condition is a visual feature, the image feature corresponding to the restoration generation condition is a visual feature and a semantic feature, and the image feature corresponding to the imitation generation condition is a visual feature, a semantic feature and a style feature; input the initial image into a conditional encoder to obtain the image feature corresponding to the at least one generation condition.
3. The method according to claim 2, wherein the conditional encoder comprises a text recognition encoder, and the image features comprise visual features; and the step of inputting the initial image into the conditional encoder to obtain the image features corresponding to the at least one generation condition comprises: Inputting the initial image into the text recognition encoder to obtain initial visual features of the initial image, and based on the initial visual features, obtaining image block indexes of the initial image; performing embedding coding on the image block indexes to obtain embedded visual features of the initial image; Pooling is performed on the initial visual features and the embedded visual features to obtain visual features of the initial image.
4. The method according to claim 2, wherein the initial image carries initial text information, the conditional encoder comprises a text recognition classifier, and the image features comprise semantic features; and the step of inputting the initial image into the conditional encoder to obtain the image features corresponding to the at least one generation condition comprises: The step of inputting the initial image into the conditional encoder to obtain image features corresponding to the at least one generation condition includes: inputting the initial text information into the text recognition classifier to obtain semantic features of the initial image, wherein the semantic features are obtained by processing an initial text sequence by the text recognition classifier, and the initial text sequence is obtained by processing encoding features of the initial text information by the text recognition classifier.
5. The method according to claim 2, wherein the initial image carries initial style information, the conditional encoder comprises a style encoder, and the image features comprise style features; and the step of inputting the initial image into the conditional encoder to obtain the image features corresponding to the at least one generation condition comprises: The initial style information is input into the style encoder to obtain the style features of the initial image.
6. The method according to claim 2, wherein the step of inputting the initial image and the image features into a diffusion generation model to obtain noise data corresponding to the initial image comprises: The image features corresponding to the initial image and the at least one generation condition are input into a diffusion generation model to obtain noise data corresponding to the initial image under each generation condition. 22 7. The method according to claim 1, wherein the target text image comprises a first target text image and a second target text image; and generating the target text image corresponding to the initial image according to the initial image and the noise data comprises: According to the initial image and the noise data, a first target text image corresponding to the initial image is generated; the first target text image is used as the updated initial image, and the step of inputting the initial image into a conditional encoder to obtain image features of the initial image is returned to generate a second target text image corresponding to the initial image.
8. The method according to claim 1, after generating a target text image corresponding to the initial image according to the initial image and the noise data, further comprising: Receiving text image generation requirement information sent by a user; In the case where the target text image does not meet the image generation requirement information, the target text image is used as the updated initial image, and the step of inputting the initial image into the conditional encoder to obtain the image features of the initial image is returned to be executed until the target text image meets the image generation requirement information to obtain the updated target text image.
9. The method according to claim 1, wherein the conditional encoder comprises a text recognition encoder and a text recognition classifier, and the training method of the conditional encoder comprises: Acquire a training set, wherein the training set includes multiple training text images, and the training text images carry classification labels; extract a first training text image from the training set, wherein the first training text image is any one of the multiple training text images; input the first training text image into the text recognition encoder, and use the output of the text recognition encoder as the input of the text recognition classifier to obtain a first predicted classification result of the first training text image; calculate an encoding loss value according to the first predicted classification result and a first classification label carried by the first training text image; adjust the parameters of the text recognition encoder and the text recognition classifier according to the encoding loss value, and return to execute the step of extracting the first training text image from the training set until a first preset stop condition is reached to obtain a conditional encoder with training completed.
10. The method according to claim 1, wherein the training method of the diffusion generation model comprises: A sample set is obtained, wherein the sample set includes multiple sample text images, and the sample text images carry sample image features; a first sample text image is extracted from the sample set, wherein the first sample text image is any one of the multiple sample text images; a first sample noise is added to the first sample text image to obtain a first noise sample image; the first noise sample image and the first sample image feature carried by the first sample text image are input into a diffusion generation model to obtain a first predicted noise; a noise loss value is calculated based on the first sample noise and the first predicted noise; a model parameter of the diffusion generation model is adjusted according to the noise loss value, and the step of extracting the first sample text image from the sample set is returned to execute until a second preset stop condition is reached to obtain a trained diffusion generation model.
11. A method for generating a handwritten text image, comprising: Receiving a handwritten text image generation request sent by a user, wherein the handwritten text image generation request carries an initial image and initial style information of the initial image; inputting the initial image and the initial style information into a conditional encoder to obtain visual features, semantic features, and style features of the initial image; The initial image, the visual features, the semantic features and the style features are input into a diffusion generation model to obtain noise data corresponding to the initial image, wherein the diffusion generation model is trained based on a sample text image, sample image features of the sample text image and a noise sample image, the noise sample image is obtained by adding sample noise to the sample text image, and the sample image features include sample visual features, sample semantic features and sample style features; based on the initial image and the noise data, a target handwritten text image corresponding to the initial image is generated; and the target handwritten text image corresponding to the initial image is sent to the user.
12. A diffusion generation model training method, applied to a cloud-side device, comprising: A sample set is obtained, wherein the sample set includes multiple sample text images, the sample text images carry sample image features, and the sample image features include at least one of sample visual features and sample semantic features; a first sample text image is extracted from the sample set, wherein the first sample text image is any one of the multiple sample text images; a first sample noise is added to the first sample text image to obtain a first noise sample image; the first noise sample image and the first sample image feature carried by the first sample text image are input into a diffusion generation model to obtain a first predicted noise; a noise loss value is calculated based on the first sample noise and the first predicted noise; a model parameter of the diffusion generation model is adjusted based on the noise loss value, and the step of extracting the first sample text image from the sample set is returned to execute until a second preset stop condition is reached to obtain the model parameters of the diffusion generation model that has been trained; and the model parameters of the diffusion generation model are sent to a terminal side device.
13. A computing device, comprising: Memory and processor; The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 10 or claim 11 or claim 12 are implemented.
14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 10 or claim 11 or claim 12.
Citation Information
Patent Citations
Text generation image method based on cross-modal similarity and generative adversarial network
CN110490946A
Text image generation method and device, terminal and computer readable storage medium
CN115438210A
Text image denoising method, device and equipment based on image generation model
CN116051418A
Image processing method and device
CN116433501A
Adversarial network for transforming handwritten text
US20210110205A1
Cited By
Image restoration method and device, electronic equipment and computer readable storage medium
CN119379564A
Image recognition method for prompting tuning based on depolarization variation inference
CN119418137A
An image recognition method based on debiased variational inference hint tuning
CN119418137B
Extremely high dynamic range imaging method and device based on event camera assistance, equipment and medium
CN119743677A
Image filling method and device, electronic equipment and storage medium
CN119850474A
