Figure generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a text map generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an initial U-shaped neural network model used for image denoising processing in a text map generation process, and the initial U-shaped neural network model comprises a plurality of groups of attention structures; replacing the multiple groups of attention structures with a target model structure with linear complexity; performing model distillation on the U-shaped neural network model after the structure replacement based on the initial U-shaped neural network model to obtain a target U-shaped neural network model; and according to the target U-shaped neural network model, carrying out image denoising on the text map to obtain a target image. According to the embodiment of the invention, the target model structure with linear complexity is adopted to replace multiple groups of attention structures, the calculation time complexity of the attention module is reduced to linear complexity, and the target U-shaped neural network model is obtained, so that the text map can generate the image with the resolution greater than 2k.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a method and device for generating a text image, an electronic device, and a storage medium. Background Art

[0002] The text-based image task is a task of generating images based on natural language text. The goal is to let the model generate images that match the input description. This task requires the model to not only recognize specific objects and scenes in the text, but also grasp abstract features such as style and emotion, and convert text information into potential visual features to generate corresponding images. The text-based image model can help designers quickly generate sketches and design concepts for use in game and film production, advertising and marketing, education and scientific research, and has broad application prospects.

[0003] The current image generation process usually uses the Unet model containing multiple groups of attention structures to achieve image denoising. The computational time complexity of the attention module makes it impossible to generate images with a resolution greater than 2k based on limited video memory. Summary of the invention

[0004] In view of the above problems, a method and device for providing a text map, an electronic device, and a storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A Vincent diagram method, the method comprising:

[0006] Acquire an initial U-shaped neural network model for image denoising in a text image process, wherein the initial U-shaped neural network model includes multiple groups of attention structures;

[0007] Replacing the multiple groups of attention structures with a target model structure having linear complexity;

[0008] Based on the initial U-shaped neural network model, model distillation is performed on the U-shaped neural network model after the replacement structure to obtain a target U-shaped neural network model;

[0009] The target image is denoised by performing image denoising on the Wensheng image according to the target U-shaped neural network model to obtain a target image.

[0010] Optionally, performing model distillation on the U-shaped neural network model after the replacement structure based on the initial U-shaped neural network model includes:

[0011] Acquire training sample data corresponding to the initial U-shaped neural network model and first noise estimation data corresponding to the training sample data;

[0012] The training sample data is input into the U-shaped neural network model after the replacement structure to obtain second noise estimation data;

[0013] The first noise estimation data is used as the expected value of image denoising of the target U-type neural network model, and model parameters of the U-type neural network model after the replacement structure are adjusted based on the first noise estimation data and the second noise estimation data to obtain the target U-type neural network model.

[0014] Optionally, the method of taking the first noise estimation data as the expected value of image denoising of the target U-type neural network model, and adjusting the model parameters of the U-type neural network model after the replacement structure based on the first noise estimation data and the second noise estimation data to obtain the target U-type neural network model includes:

[0015] Constructing a loss function of the first noise estimation data and the second noise estimation data by taking the first noise estimation data as an expected value of image denoising of the target U-shaped neural network model;

[0016] Based on the loss function, the model parameters of the U-shaped neural network model after the replacement structure are adjusted to obtain a target U-shaped neural network model.

[0017] Optionally, the loss function is the square of the L2 norm of the difference between the second noise estimation data and the first noise estimation data.

[0018] Optionally, performing image denoising of the Vincent graph according to the target U-shaped neural network model includes:

[0019] A pre-trained variational autoencoder is used to convert the initial image into latent space features;

[0020] Adding Gaussian noise to the latent space feature to obtain a latent space noise feature;

[0021] Acquire text information input by a user, and encode the text information into text features;

[0022] Inputting the text features and the latent space noise features into a target U-shaped neural network model to perform image denoising, and outputting denoising features;

[0023] A target image is generated based on the denoising feature decoding.

[0024] Optionally, encoding the text information into text features includes:

[0025] Obtaining a target language model for text encoding;

[0026] The text information is input into the target language model, and an encoder in the target language model is used to encode the text to obtain text features.

[0027] Optionally, the target model structure is a Mamba2 structure based on a state space model.

[0028] A Wensheng graph device, comprising:

[0029] An initial U-shaped neural network model acquisition module is used to acquire an initial U-shaped neural network model for image denoising in the process of performing Wenshengtu, wherein the initial U-shaped neural network model includes multiple groups of attention structures;

[0030] A structure replacement module, used to replace the multiple groups of attention structures with a target model structure having linear complexity;

[0031] A model distillation module, used for performing model distillation on the U-shaped neural network model after the replacement structure based on the initial U-shaped neural network model to obtain a target U-shaped neural network model;

[0032] The target image generation module is used to perform image denoising of the Wensheng image according to the target U-shaped neural network model to obtain a target image.

[0033] An electronic device, characterized in that it includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the above-mentioned Wensheng diagram method is implemented.

[0034] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned Vincent diagram method is implemented.

[0035] The embodiments of the present invention have the following advantages:

[0036] In an embodiment of the present invention, an initial U-type neural network model for image denoising in the process of performing Wensheng graph processing is obtained, and the initial U-type neural network model includes multiple groups of attention structures; the multiple groups of attention structures are replaced by a target model structure with linear complexity; the U-type neural network model after the replacement structure is model distilled based on the initial U-type neural network model to obtain a target U-type neural network model; the image denoising of the Wensheng graph is performed according to the target U-type neural network model to obtain a target image, and it can be realized that the multiple groups of attention structures are replaced by a target model structure with linear complexity, so that the computational time complexity of the attention module can be reduced to linear complexity, and the target U-type neural network model is obtained, so that the Wensheng graph can generate images with a resolution greater than 2k. At the same time, by generating the target U-type neural network model by model distillation, the knowledge learned in the past can be effectively utilized to improve the efficiency and performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0038] Figure 1 is a flowchart of a method for producing a text map according to an embodiment of the present invention;

[0039] Figure 2 is a flowchart of another method for producing a text map according to an embodiment of the present invention;

[0040] Figure 3 is a flowchart of another method for producing a text map according to an embodiment of the present invention;

[0041] Figure 4 It is a schematic diagram of a process of a Vincent diagram provided by an embodiment of the present invention;

[0042] Figure 5 It is a schematic diagram of the structure of a Wensheng graph device provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Reference Figure 1 , shows a flowchart of a method for a text map provided by an embodiment of the present invention, which may specifically include the following steps:

[0045] Step S101, obtaining an initial U-shaped neural network model for image denoising in a text image process, wherein the initial U-shaped neural network model includes multiple groups of attention structures;

[0046] In practical applications, the text graph processing process may include image feature extraction, image feature noise addition, image feature denoising, decoding and generating images, etc. Among them, a U-type neural network model can be used for image denoising, namely the Unet model. The Unet model is a model structure that uses a symmetrical U-shaped architecture and jump connections, which can effectively capture the global and local features of the image. The initial U-type neural network model in the current text graph model denoising process may include multiple groups of attention structures, each of which includes a self-attention block and a cross-attention block. The self-attention block and the cross-attention block have computational time complexity. Under the condition of limited video memory, it is difficult to generate images with a resolution greater than 2k.

[0047] The initial U-type neural network model in the embodiment of the present invention is the Unet model that has been trained in the process of the Wensheng graph. Specifically, the Unet model mainly includes a self-attention block and a cross-attention block. These two modules form a group, and a total of N groups constitute the Unet network θ. The network predicts the loss L noise To train:

[0048]

[0049] Among them, L noise is the noise prediction loss, ∈ is the Gaussian noise added once in the Vincent graph process, z t is the latent space noise feature in the input Unet model, t is the number of times Gaussian noise is added, y is the text feature in the image denoising process, ∈ θ is the noise feature z in the latent space based on the Unet model t , the number of times noise is added t, and the estimated noise when the text feature y is input.

[0050] In the Unet network, the self-attention module and the cross-attention module have the same structure. The difference between the self-attention module and the cross-attention module is that the self-attention module only uses the image feature z for feature self-attention information mining, while the cross-attention module uses the image feature z and the text feature y for cross-attention information mining.

[0051] Assume that the image features and text features are Weight Parameters Where n is the number of tokens, d is the feature dimension, and d′ is the attention dimension.

[0052]

[0053] Among them, M is the attention map. When Q and K both come from image features, the above formula represents a self-attention module. When Q comes from image features and K comes from text features, the above formula is a cross-attention module.

[0054] The above-mentioned Unet model is trained with hundreds of millions of image-text pairs, so that the Unet model can accurately convert noise into image latent space features according to text information, so that the complete image can be restored only by noise and text features in the inference stage.

[0055] In the Unet model, the calculation method of Q (Query), K (Key) and V (Value) makes the calculation of a single group of attention increase exponentially. At the same time, the use of multiple groups of attention structures in the Unet model makes the calculated data dimension too high, which is not conducive to calculation in limited video memory.

[0056] Step S102, replacing multiple groups of attention structures with a target model structure having linear complexity;

[0057] After obtaining the initial U-shaped neural network model, the multiple groups of attention structures in the initial U-shaped neural network model can be improved. Specifically, the multiple groups of attention structures can be replaced by a target model structure with linear complexity, so that the computational time complexity of the attention module can be reduced to linear, so that larger resolution images can be generated based on limited video memory.

[0058] Among them, the target model structure is the Mamba2 structure based on the state space model. Among them, the state space model (State Space Model, SSM) is a mathematical model used to describe dynamic systems. It helps to analyze and predict time series data through the relationship between observed variables and hidden state variables. The Mamba2 structure of the state space model, Mamba2 is a sequence reasoning model with linear complexity, designed to efficiently process long sequence data. The design goal of Mamba2 is to achieve linear time complexity, so that it has significant computational advantages when processing large-scale sequence data. Mamba2 can achieve this goal by introducing the state space model (State Space Model).

[0059] In one embodiment of the present invention, the state space model SSM can be expressed as follows:

[0060]

[0061] Y i =Ci H i ,

[0062] Where i represents the serial number of the current token in the sequence. is the hidden state, are independent variables, and ⊙ represents the multiplication of elements one by one. The meaning of the above formula is: H i is the hidden state at the current moment, through the hidden state H at the previous moment i-1 and the current input X i Updated, A i and B i It is a parameter related to the current moment and is used to adjust the update method of the hidden state. i is the output at the current moment, through the current hidden state H i and parameter C i Calculated.

[0063] Then the above formula can be expressed as:

[0064]

[0065] in, is an n×n lower triangular matrix, This structure is Mamba2, which has linear complexity. Combined, the computation of the state-space model can be simplified to linear complexity. The elements in are elements That is A k The cumulative product of .

[0066] Step S103, performing model distillation on the U-shaped neural network model after the replacement structure based on the initial U-shaped neural network model to obtain a target U-shaped neural network model;

[0067] After the structural replacement is completed, the replaced U-shaped neural network model may not achieve the pre-set image denoising effect. Therefore, the U-shaped neural network model after the replacement structure can be distilled with the image denoising effect corresponding to the previously trained initial U-shaped neural network model as the goal. Model distillation is a technology used for model compression and acceleration. It transfers the knowledge of a complex, large, and well-trained teacher model (which achieves high accuracy on the target task) to a simpler, smaller student model, so that the student model has lower computational complexity and faster reasoning speed while maintaining high performance. During the model distillation process, the student model is usually smaller than the teacher model and has fewer parameters, thereby reducing the model's storage space and computing resource requirements. At the same time, due to its small size and faster reasoning speed, the student model is suitable for running on resource-constrained devices (such as mobile devices, embedded systems, etc.). In addition, by learning the knowledge of the teacher model, the student model can achieve model compression and acceleration while maintaining high performance.

[0068] In the embodiment of the present invention, the initial U-shaped neural network model is the teacher model, and the U-shaped neural network model after the replacement structure is the student model. The knowledge of the original Unet is distilled into the Unet structure replaced by the target model structure by distillation, thereby effectively utilizing the knowledge learned in the past. After large-scale distillation, the new Unet structure θ′ has the same denoising ability as the original Unet structure θ, and the structure has a linear time complexity, so it can be used to generate ultra-high resolution images.

[0069] Step S104, performing image denoising on the Wensheng image according to the target U-shaped neural network model to obtain a target image.

[0070] After the U-shaped neural network model is generated, the target U-shaped neural network model can be applied to the image generation process to replace the original initial U-shaped neural network model for image denoising, thereby obtaining a high-resolution target image greater than 2K.

[0071] In an embodiment of the present invention, an initial U-type neural network model for image denoising in the process of performing Wensheng graph processing is obtained, and the initial U-type neural network model includes multiple groups of attention structures; the multiple groups of attention structures are replaced by a target model structure with linear complexity; the U-type neural network model after the replacement structure is model distilled based on the initial U-type neural network model to obtain a target U-type neural network model; the image denoising of the Wensheng graph is performed according to the target U-type neural network model to obtain a target image, and it can be realized that the multiple groups of attention structures are replaced by a target model structure with linear complexity, and the computational time complexity of the attention module is reduced to linear complexity to obtain a target U-type neural network model, so that the Wensheng graph can generate an image with a resolution greater than 2k. At the same time, by generating the target U-type neural network model by model distillation, the knowledge learned in the past can be effectively utilized to improve the efficiency and performance of the model.

[0072] Reference Figure 2 , shows a flowchart of another method for producing a text map according to an embodiment of the present invention, which may include the following steps:

[0073] Step S201, obtaining an initial U-shaped neural network model for image denoising in a text-generated image process, wherein the initial U-shaped neural network model includes multiple groups of attention structures;

[0074] Step S202, replacing multiple groups of attention structures with a target model structure having linear complexity;

[0075] Step S203, obtaining training sample data corresponding to the initial U-shaped neural network model and first noise estimation data corresponding to the training samples;

[0076] The training sample data of the initial U-shaped neural network model may include image features, noise, the number of noise additions, and text features.

[0077] Step S204, inputting the training sample data into the U-shaped neural network model after the replacement structure to obtain second noise estimation data;

[0078] According to the training process of the initial U-shaped neural network model, the training sample data is input into the U-shaped neural network model after the replacement structure to perform noise estimation to obtain second noise estimation data.

[0079] Step S205, taking the first noise estimation data as the expected value of image denoising of the target U-type neural network model, adjusting the model parameters of the U-type neural network model after the replacement structure based on the first noise estimation data and the second noise estimation data to obtain the target U-type neural network model.

[0080] In practical applications, since the initial U-type neural network model is a trained and usable model, the image denoising ability of the initial U-type neural network model meets the requirements of the Vincent diagram. Therefore, the first noise estimation data can be used as the expected value of the image denoising process of the target U-type neural network model to evaluate the denoising ability of the U-type neural network model after the replacement structure, and then the denoising ability of the U-type neural network model after the replacement structure is optimized by modifying the model parameters in the target model structure, so that its denoising ability can reach the denoising ability corresponding to the initial U-type neural network model.

[0081] In an embodiment of the present invention, the unreplaced part of the U-shaped neural network model maintains the previous parameters, and only the model parameters of the target model structure of the replaced part are adjusted, which can reduce the amount of model parameter modification. In an embodiment of the present invention, step S205 may include the following sub-steps:

[0082] Sub-step S11, constructing a loss function of the first noise estimation data and the second noise estimation data with the first noise estimation data as the expected value of image denoising of the target U-type neural network model;

[0083] The loss function is the square of the L2 norm of the difference between the second noise estimation data and the first noise estimation data.

[0084] The model is trained on sample data (z t ,t,y) under the second noise estimation data;∈ θ (z t ,t,y) represents the initial U-type neural network model in the training sample data (z t ,t,y) under the first noise estimation data.

[0086] Sub-step S12, based on the loss function, adjust the model parameters of the U-shaped neural network model after the replacement structure to obtain the target U-shaped neural network model.

[0087] The model parameters of the U-shaped neural network model after the replacement structure are adjusted based on the value calculated by the loss function, so that the second noise estimation data of the adjusted model approaches the first noise estimation data.

[0088] In one embodiment of the present invention, during the model distillation process, a loss function threshold can be set. When the loss function value of the model is within the loss function threshold range, the model distillation can be stopped and the current model is determined to be the target U-type neural network model. If the loss function value of the model is not within the loss function threshold range, the model distillation continues.

[0089] Step S206, performing image denoising on the Wensheng image according to the target U-shaped neural network model to obtain a target image.

[0090] In an embodiment of the present invention, a target model structure with linear complexity can be used to replace multiple groups of attention structures, and the computational time complexity of the attention module can be reduced to linear complexity to obtain a target U-type neural network model, so that the Vincent graph can generate images with a resolution greater than 2k. At the same time, by comparing the noise estimate of the trained U-type neural network model with the noise estimate of the replaced U-type neural network model, a target U-type neural network model can be generated, thereby effectively utilizing the knowledge learned in the past and improving the efficiency and performance of the model.

[0091] Reference Figure 3 , shows a flowchart of another method for producing a text map according to an embodiment of the present invention, which may include the following steps:

[0092] Step S301, obtaining an initial U-shaped neural network model for image denoising in a text image process, wherein the initial U-shaped neural network model includes multiple groups of attention structures;

[0093] Step S302, replacing multiple groups of attention structures with a target model structure having linear complexity;

[0094] Step S303, performing model distillation on the U-shaped neural network model after the replacement structure based on the initial U-shaped neural network model to obtain a target U-shaped neural network model;

[0095] Step S304, using a pre-trained variational autoencoder to convert the initial image into latent space features;

[0096] The initial image can be converted into latent space features through a pre-trained variational autoencoder. This process includes loading a pre-trained VAE (Variational AutoEncoder) model, preprocessing the input image, encoding the image into the latent space, and sampling the latent space features.

[0097] VAE consists of two parts: an encoder and a decoder. The encoder can map input data (such as images) to latent variables in the latent space (such as mean and variance); the decoder can reconstruct the input data from the latent variables in the latent space.

[0098] The pre-trained VAE used in the embodiment of the present invention has been trained on a large amount of data, and can effectively map the input data to the latent space and generate high-quality samples from the latent space. The pre-trained VAE can be directly used for feature extraction tasks without retraining. The pre-trained model can be obtained from a public model library or the appendix of a research paper.

[0099] In practical applications, before the initial image is input into the loaded VAE model, the initial image can be preprocessed to meet the input requirements of the VAE model. Preprocessing can include operations such as image scaling and normalization. Then the preprocessed initial image is input into the VAE model, and the encoder part of the VAE is used to encode the preprocessed image into the latent space to obtain the latent variables (mean and variance). The reparameterization technique is used to sample the latent variables from the mean and variance.

[0100] Step S305, adding Gaussian noise to the latent space feature to obtain the latent space noise feature;

[0101] Among them, Gaussian noise is a kind of random noise, and its value obeys normal distribution (i.e. Gaussian distribution). The purpose of adding noise is to introduce randomness so that the generated image has more changes and diversity.

[0102] Step S306, obtaining text information input by the user, and encoding the text information into text features;

[0103] In one embodiment of the present invention, encoding text information into text features includes: obtaining a target language model for text encoding; inputting the text information into the target language model, using an encoder in the target language model to perform text encoding, and obtaining text features. Compared with the existing text graph generation process in which two or more encoders are used to perform text encoding, the structure of the model is simplified.

[0104] In practical applications, the target language model can adopt the GLM4 large language model that supports multiple languages. Different from the common CLIP encoder (Contrastive Language-Image Pretraining) and T5 encoder (Text-To-Text Transfer Transformer), GLM4 (General Language Model 4) can greatly improve the representation ability of text features.

[0105] Step S307, inputting the text features and latent space noise features into the target U-shaped neural network model to perform image denoising, and outputting denoising features;

[0106] In the process of image denoising, directional denoising is achieved according to text features.

[0107] Step S308: Generate a target image based on denoising feature decoding.

[0108] The decoder of the variational autoencoder is used to decode the denoising features to obtain the target image.

[0109] In the embodiment of the present invention, a target model structure with linear complexity can be used to replace multiple groups of attention structures, and the computational time complexity of the attention module can be reduced to linear complexity to obtain a target U-shaped neural network model, so that the Vincent graph can generate images with a resolution greater than 2k. At the same time, by generating the target U-shaped neural network model through model distillation, the knowledge learned in the past can be effectively utilized to improve the model efficiency and performance.

[0110] Reference Figure 4 , shows a schematic diagram of a Wensheng diagram process in an embodiment of the present invention, which may include the following process:

[0111] Get Figure 4 The initial image on the left is input into the encoding module (VAE Encoder) of the VAE encoder, and noise is added to generate the latent space noise feature Zt.

[0112] The user inputs the text information "Three cows are grazing leisurely under the white birch trees. The leaves of the birch trees are yellow, the grass is also yellow, and there is a white cow in the distance. The whole picture creates a tranquil atmosphere", and the text information is input into the GLM4 large language model to generate text encoding, and then the latent space noise feature Zt and the latent space noise feature Zt image denoising module can be used for image denoising. Among them, the upper part is the original Unet model, which is composed of multiple groups of self-attention blocks and cross-attention blocks. The lower part is the Mamba2 structure based on the state space model (State Space Model, SSM) replaces the attention structure, and the model is distilled to obtain the target Unet model. After the target Unet model, the latent space noise feature Zt can be denoised according to the text features, and then the denoised features are obtained. The denoised features are input into the decoding module (VAE Decoder) of the VAE encoder to obtain the generated target image ( Figure 4 right image).

[0113] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0114] Reference Figure 5, shows a schematic diagram of the structure of a Wensheng graph device provided by an embodiment of the present invention, which may specifically include the following modules:

[0115] An initial U-shaped neural network model acquisition module 501 is used to acquire an initial U-shaped neural network model for image denoising in a text-generated image process, wherein the initial U-shaped neural network model includes multiple groups of attention structures;

[0116] A structure replacement module 502, used to replace the multiple groups of attention structures with a target model structure having linear complexity;

[0117] A model distillation module 503 is used to perform model distillation on the U-shaped neural network model after the replacement structure based on the initial U-shaped neural network model to obtain a target U-shaped neural network model;

[0118] The target image generation module 504 is used to perform image denoising of the Wensheng image according to the target U-shaped neural network model to obtain a target image.

[0119] In one embodiment of the present invention, the model distillation module 503 may include the following submodules:

[0120] A sample acquisition submodule, used to acquire training sample data corresponding to the initial U-shaped neural network model and first noise estimation data corresponding to the training sample;

[0121] A second noise estimation determination submodule is used for inputting the training sample data into the U-shaped neural network model after the replacement structure to obtain second noise estimation data;

[0122] A model parameter adjustment submodule is used to use the first noise estimation data as the expected value of image denoising of the target U-type neural network model, and adjust the model parameters of the U-type neural network model after the replacement structure based on the first noise estimation data and the second noise estimation data to obtain the target U-type neural network model.

[0123] In one embodiment of the present invention, the model parameter adjustment submodule may include the following units:

[0124] A loss function construction unit, configured to construct a loss function of the first noise estimation data and the second noise estimation data by taking the first noise estimation data as an expected value of image denoising of the target U-shaped neural network model;

[0125] The U-type neural network model determination unit is used to adjust the model parameters of the U-type neural network model after the replacement structure based on the loss function to obtain a target U-type neural network model.

[0126] In one embodiment of the present invention, the loss function is the square of the L2 norm of the difference between the second noise estimation data and the first noise estimation data.

[0127] In an embodiment of the present invention, the target image generation module 504 may include:

[0128] The latent space feature conversion submodule is used to convert the initial image into latent space features using a pre-trained variational autoencoder;

[0129] A latent space noise feature submodule, used for adding Gaussian noise to the latent space feature to obtain a latent space noise feature;

[0130] A text information encoding submodule, used to obtain text information input by a user and encode the text information into text features;

[0131] A denoising submodule, used for inputting the text features and latent space noise features into a target U-shaped neural network model to perform image denoising and output denoising features;

[0132] A decoding submodule is used to generate a target image based on the denoising feature decoding.

[0133] In one embodiment of the present invention, the text information encoding submodule may include the following units:

[0134] A target speech model acquisition unit, used to acquire a target language model for text encoding;

[0135] The text feature encoding unit is used to input the text information into the target language model, use an encoder in the target language model to perform text encoding, and obtain text features.

[0136] In one embodiment of the present invention, the target model structure is a Mamba2 structure based on a state space model.

[0137] In an embodiment of the present invention, an initial U-type neural network model for image denoising in the process of performing Wensheng graph processing is obtained, and the initial U-type neural network model includes multiple groups of attention structures; the multiple groups of attention structures are replaced by a target model structure with linear complexity; the U-type neural network model after the replacement structure is model distilled based on the initial U-type neural network model to obtain a target U-type neural network model; the image denoising of the Wensheng graph is performed according to the target U-type neural network model to obtain a target image, so that the target model structure with linear complexity is used to replace the multiple groups of attention structures, and the computational time complexity of the attention module is reduced to linear complexity to obtain the target U-type neural network model, so that the Wensheng graph can generate images with a resolution greater than 2k. At the same time, by generating the target U-type neural network model by model distillation, the knowledge learned in the past can be effectively utilized to improve the efficiency and performance of the model.

[0138] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the above-mentioned image generation method is implemented.

[0139] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image generation method is implemented.

[0140] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0141] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0142] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0143] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0146] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0147] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0148] The above provides a detailed introduction to a text image method and device, electronic device, and storage medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A Vincent diagram method, characterized in that: The method comprises: Acquire an initial U-shaped neural network model for image denoising in a text image process, wherein the initial U-shaped neural network model includes multiple groups of attention structures; Replacing the multiple groups of attention structures with a target model structure having linear complexity; Based on the initial U-shaped neural network model, model distillation is performed on the U-shaped neural network model after the replacement structure to obtain a target U-shaped neural network model; The target image is denoised by performing image denoising on the Wensheng image according to the target U-shaped neural network model to obtain a target image.

2. The method according to claim 1, characterized in that The performing model distillation on the U-shaped neural network model after the replacement structure based on the initial U-shaped neural network model includes: Acquire training sample data corresponding to the initial U-shaped neural network model and first noise estimation data corresponding to the training sample data; Inputting the training sample data into the U-shaped neural network model after the replacement structure to obtain second noise estimation data; Taking the first noise estimation data as the expected value of image denoising of the target U-type neural network model, adjusting the model parameters of the U-type neural network model after the replacement structure based on the first noise estimation data and the second noise estimation data, to obtain the target U-type neural network model.

3. The method according to claim 2, characterized in that The method of taking the first noise estimation data as the expected value of the image denoising of the target U-type neural network model, and adjusting the model parameters of the U-type neural network model after the replacement structure based on the first noise estimation data and the second noise estimation data to obtain the target U-type neural network model includes: Constructing a loss function of the first noise estimation data and the second noise estimation data by taking the first noise estimation data as an expected value of image denoising of the target U-shaped neural network model; Based on the loss function, the model parameters of the U-shaped neural network model after the replacement structure are adjusted to obtain a target U-shaped neural network model.

4. The method according to claim 3, characterized in that The loss function is the square of the L2 norm of the difference between the second noise estimation data and the first noise estimation data.

5. The method according to any one of claims 1 to 4, characterized in that The performing image denoising of the Wensheng graph according to the target U-shaped neural network model includes: A pre-trained variational autoencoder is used to convert the initial image into latent space features; Adding Gaussian noise to the latent space feature to obtain a latent space noise feature; Acquire text information input by a user, and encode the text information into text features; Inputting the text features and the latent space noise features into a target U-shaped neural network model to perform image denoising, and outputting denoising features; A target image is generated based on the denoising feature decoding.

6. The method according to claim 5, characterized in that The step of encoding the text information into text features comprises: Obtaining a target language model for text encoding; The text information is input into the target language model, and an encoder in the target language model is used to encode the text to obtain text features.

7. The method according to claim 1 or 2 or 3 or 4 or 6, characterized in that: The target model structure is a Mamba2 structure based on a state space model.

8. A Wensheng map device, characterized in that: The device comprises: An initial U-shaped neural network model acquisition module is used to acquire an initial U-shaped neural network model for image denoising in the process of performing Wenshengtu, wherein the initial U-shaped neural network model includes multiple groups of attention structures; A structure replacement module, used to replace the multiple groups of attention structures with a target model structure having linear complexity; A model distillation module, used for performing model distillation on the U-shaped neural network model after the replacement structure based on the initial U-shaped neural network model to obtain a target U-shaped neural network model; The target image generation module is used to perform image denoising of the Wensheng image according to the target U-shaped neural network model to obtain a target image.

9. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the Vincent diagram method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for implementing the Vincent diagram according to any one of claims 1 to 7 is implemented.

Images