Model processing method, apparatus, product, device, and medium

Abstract

The application discloses a model processing method, device, product, equipment and medium, and the method comprises the steps of obtaining a sample data pair, the sample data pair comprising sample text and a sample image matching the text content of the sample text; calling a noise reduction model to perform N-step noise reduction processing on a first noise image generated by the sample image based on the sample text, to generate a noise reduction feature corresponding to each step of noise reduction processing; obtaining M noise reduction features from the N noise reduction features for loss feedback of the noise reduction model; generating feedback loss for the noise reduction model based on the M noise reduction features; and correcting the model parameters of the noise reduction model based on the generated M feedback losses to obtain a trained noise reduction model, wherein the trained noise reduction model can generate an image matching the text content of input text. The application can improve the accuracy of correcting the model parameters of the noise reduction model, thereby improving the effect of image generation by the trained noise reduction model.

Description

Technical Field

[0001] This application relates to the technical field of artificial intelligence, and more particularly to a model processing method, apparatus, product, device, and medium. Background Technology

[0002] In the field of artificial intelligence, users can input a string of text descriptions, and the backend can call upon a trained neural network to generate an image that matches the text content of the user's input. However, during the training process of the neural network, the network parameters are only corrected by comparing the generated image with the reference sample image. This method of parameter correction is relatively simple and the overall correction process is rather general. It is easy for the network parameters to not be accurately corrected, resulting in poor image quality generated by the trained neural network.

[0003] Therefore, improving the accuracy of correcting the network parameters of neural networks, thereby enhancing the image generation performance of the trained neural networks, is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a model processing method, apparatus, product, device, and medium that can improve the accuracy of correcting the model parameters of a denoising model, thereby improving the image generation effect through a trained denoising model.

[0005] This application provides a model processing method, which includes:

[0006] Obtain sample data pairs, which include sample text and sample images that match the text content of the sample text;

[0007] The sample image is noise-added to generate a first noise map, and the noise reduction model is called to perform N-step noise reduction on the first noise map based on the sample text, generating noise reduction features corresponding to each step of the noise reduction process, where N is a positive integer.

[0008] From the N noise reduction features corresponding to the N-step noise reduction process, obtain M noise reduction features for loss feedback of the noise reduction model, where M is a positive integer and M is less than N;

[0009] Loss feedback processing is performed based on M denoising features to generate a feedback loss for the denoising model. Each of the M denoising features corresponds to a type of feedback loss.

[0010] Based on the generated M feedback losses, the model parameters of the denoising model are corrected to obtain a trained denoising model. The trained denoising model is used to generate an image that matches the text content of the input text.

[0011] This application provides a model processing apparatus, which includes:

[0012] The first acquisition module is used to acquire sample data pairs, which include sample text and sample images that match the text content of the sample text.

[0013] The noise reduction module is used to add noise to the sample image, generate a first noise map, and call the noise reduction model to perform N-step noise reduction on the first noise map based on the sample text, generating noise reduction features corresponding to each step of the noise reduction process, where N is a positive integer.

[0014] The second acquisition module is used to acquire M denoising features from the N denoising features corresponding to the N-step denoising process, which are used for loss feedback of the denoising model. M is a positive integer and M is less than N.

[0015] The feedback module is used to perform loss feedback processing based on M denoising features, and generate a feedback loss for the denoising model. Each of the M denoising features generates a corresponding type of feedback loss.

[0016] The correction module is used to correct the model parameters of the denoising model based on the generated M feedback losses, so as to obtain the trained denoising model. The trained denoising model is used to generate an image that matches the text content of the input text.

[0017] Optionally, the noise reduction module may add noise to the sample image to generate a first noise map in the following ways:

[0018] The image encoder is invoked to perform feature encoding processing on the sample image, generating the image encoding features of the sample image;

[0019] The image encoding features of the sample image are noise-added to generate an initial noise map;

[0020] The noise reduction model is invoked to perform feature mapping on the initial noise map, generating the first noise map.

[0021] Optionally, the second acquisition module obtains M denoising features from the N denoising features corresponding to the N denoising steps in the denoising process, which are used for loss feedback on the denoising model, in the following ways:

[0022] Get the set first step number i and second step number j; i and j are both positive integers, and i and j are less than or equal to N respectively;

[0023] The noise reduction features corresponding to the i-th step and the j-th step are determined as M noise reduction features from the N noise reduction features;

[0024] Where i is less than j, the denoising feature corresponding to the i-th denoising step is used to generate the feedback loss of the image-text matching type, and the denoising feature corresponding to the j-th denoising step is used to generate the feedback loss of the image aesthetic type.

[0025] Optionally, the loss feedback processing includes image-text matching processing. The feedback module performs loss feedback processing based on M denoising features, generating feedback loss for the denoising model in the following ways:

[0026] The image decoder is invoked to perform feature decoding on the denoising features corresponding to the denoising process in step i, and the first predicted image is generated.

[0027] Perform image-text matching processing on the first predicted image and the sample text to obtain the image-text matching degree between the first predicted image and the sample text.

[0028] Image-text matching loss is used to generate a noise reduction model based on image-text matching degree; image-text matching loss is a type of loss that belongs to the image-text matching property type.

[0029] Based on the image-text matching loss, the feedback loss for the denoising model in the i-th step of the denoising process is generated.

[0030] Optionally, the feedback module performs image-text matching processing on the first predicted image and the sample text to obtain the image-text matching degree between the first predicted image and the sample text, including the following methods:

[0031] The image processing model is invoked to perform feature transformation processing on the first predicted image, generating the transformed image features of the first predicted image;

[0032] The image processing model is invoked to perform feature transformation on the sample text, generating transformed text features of the sample text; the transformed image features and the transformed text features are in the same feature space of the image processing model;

[0033] Calculate the feature matching degree between the transformed image features and the transformed text features, and use the feature matching degree as the image-text matching degree;

[0034] The image-text matching process includes feature transformation processing of the first predicted image, feature transformation processing of the sample text, and calculation of the feature matching degree between the transformed image features and the transformed text features.

[0035] Optionally, the first noise map is obtained by adding a sample noise map to the image coding features of the sample image;

[0036] The feedback module generates the feedback loss for the denoising model in the i-th step of the denoising process based on the image-text matching loss, including:

[0037] Calculate the difference between the noise reduction feature corresponding to the i-th step noise reduction process and the first noise map, and use it as the predicted noise map corresponding to the i-th step noise reduction process.

[0038] Based on the difference between the predicted noise map and the sample noise map corresponding to the i-th step of noise reduction, the first noise prediction loss is generated.

[0039] The first noise prediction loss and the image-text matching loss are summed to generate the feedback loss for the denoising model in the i-th step of denoising processing.

[0040] Optionally, the loss feedback processing includes an evaluation of image aesthetics. The feedback module performs loss feedback processing based on M denoising features, generating a feedback loss for the denoising model in the following ways:

[0041] The image decoder is invoked to perform feature decoding on the denoising features corresponding to the j-th step of denoising, and the second predicted image is generated.

[0042] The aesthetic evaluation model is called to evaluate the aesthetics of the second predicted image, generate an aesthetic score for the second predicted image, and generate the aesthetic loss of the denoising model based on the aesthetic score; the aesthetic loss belongs to the loss of image aesthetics.

[0043] Based on the aesthetic loss, the feedback loss of the j-th step noise reduction process for the noise reduction model is generated.

[0044] Optionally, the first noise map is obtained by adding a sample noise map to the image coding features of the sample image;

[0045] The feedback module generates the feedback loss for the denoising model in the j-th step of the denoising process based on the aesthetic loss, including:

[0046] Calculate the difference between the denoised feature corresponding to the j-th denoising step and the first noise map, and use it as the predicted noise map corresponding to the j-th denoising step.

[0047] Based on the difference between the predicted noise map and the sample noise map corresponding to the j-th step of noise reduction, a second noise prediction loss is generated.

[0048] The second noise prediction loss and aesthetic loss are summed to generate the feedback loss of the noise reduction model for the j-th step of noise reduction processing.

[0049] Optionally, the correction module can correct the model parameters of the denoising model based on the generated M feedback losses to obtain the trained denoising model in the following ways:

[0050] Obtain the image generation loss of the noise reduction model;

[0051] The image generation loss is summed with each of the M feedback losses to generate the total M feedback losses.

[0052] The model parameters of the denoising model are successively corrected based on the total loss of M feedbacks to obtain the trained denoising model.

[0053] Optionally, the method by which the correction module obtains the image generation loss of the denoising model includes:

[0054] The image decoder is invoked to perform feature decoding on the denoising features corresponding to the Nth denoising step in the N-step denoising process, and a third prediction image is generated.

[0055] Based on the difference between the third predicted image and the sample image, the image generation loss of the denoising model is generated.

[0056] Optionally, the correction module may sum the image generation loss with each of the M feedback losses to generate the total M feedback losses, including:

[0057] Obtain the first weight coefficient corresponding to the image generation loss, and the second weight coefficient corresponding to each feedback loss;

[0058] Based on the first weight coefficient and the second weight coefficient corresponding to each feedback loss, the image generation loss is summed with each feedback loss to generate M total feedback losses.

[0059] Optionally, the above-mentioned model processing device further includes a fine-tuning module, which is used for:

[0060] Obtain the initial noise reduction model and add noise to the sample images to generate a second noise map;

[0061] The initial denoising model is invoked to denoise the second noise map based on the sample text, generating sample denoising features;

[0062] The image decoder is invoked to perform feature decoding on the sample denoising features, generating a sample prediction image;

[0063] Based on the difference between the predicted image and the sample image, the model parameters of the initial denoising model are corrected to obtain the denoising model.

[0064] Optionally, the above-mentioned model processing device further includes a business module, which is used for:

[0065] Obtain the image generation request submitted by the business client. The image generation request carries the target text.

[0066] The target text is input into the trained denoising model, which then generates a target image that matches the text content of the target text.

[0067] The target image is returned to the business client, enabling the business client to output the target image.

[0068] Optionally, the business module may use the trained denoising model to generate a target image that matches the text content of the target text, including:

[0069] The trained noise reduction model is invoked to randomly generate a third noise map;

[0070] The trained denoising model is invoked to denoise the third noise map based on the target text, generating the target denoising features;

[0071] The image decoder is invoked to perform feature decoding processing on the target noise reduction features, generating the target image.

[0072] This application provides a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method of this application.

[0073] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the method described in the above-mentioned aspect.

[0074] According to one aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the methods provided in the various alternative embodiments described above.

[0075] This application can acquire sample data pairs, including sample text and sample images matching the text content of the sample text; it can add noise to the sample images to generate a first noise map, and call a denoising model to perform N-step denoising on the first noise map based on the sample text, generating denoising features corresponding to each denoising step, where N is a positive integer; thus, from the N denoising features corresponding to the N-step denoising, M denoising features for loss feedback to the denoising model can be obtained, where M is a positive integer and M is less than N; this application can also perform loss feedback processing based on the M denoising features to generate feedback loss for the denoising model, where each of the M denoising features corresponds to a type of feedback loss; the model parameters of the denoising model are corrected based on the generated M feedback losses to obtain a trained denoising model, which is used to generate an image matching the text content of the input text. Therefore, the method proposed in this application can select denoising features (such as the selected M denoising features) corresponding to some steps of denoising processing in multiple steps of denoising processing of a noisy map (such as the first noisy map) (such as N steps of denoising processing) to generate different types of feedback losses for the denoising model. Thus, the model parameters of the denoising model can be more comprehensively and accurately corrected through the generated various types of feedback losses, and finally a well-trained denoising model can be obtained. The well-trained denoising model can then generate an image that matches the text content of the input text and has excellent effect. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a schematic diagram of a network architecture for generating images from text, provided in an embodiment of this application.

[0078] Figure 2 This is a schematic diagram of a scenario for training a noise reduction model according to an embodiment of this application;

[0079] Figure 3 This is a schematic flowchart of a model processing method provided in an embodiment of this application;

[0080] Figure 4 This is a schematic diagram of a neural network structure provided in an embodiment of this application;

[0081] Figure 5This is a schematic diagram illustrating the effect of the client interface of a business client provided in an embodiment of this application;

[0082] Figure 6 This is a schematic diagram of a process for generating a feedback loss of image-text matching type according to an embodiment of this application;

[0083] Figure 7 This is a schematic diagram of a scenario for generating a feedback loss of the image-text matching type provided in an embodiment of this application;

[0084] Figure 8 This is a schematic diagram of a process for generating feedback loss of image aesthetics type according to an embodiment of this application;

[0085] Figure 9 This is a schematic diagram of a scenario for feedback loss of image aesthetics type provided in an embodiment of this application;

[0086] Figure 10 This is a schematic diagram of a process for training an initial denoising model to obtain a denoising model, provided in an embodiment of this application.

[0087] Figure 11 This is a schematic diagram of an architecture for feedback training of a noise reduction model provided in an embodiment of this application;

[0088] Figure 12 This is a schematic diagram of the structure of a model processing device provided in an embodiment of this application;

[0089] Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0090] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0091] First, it should be noted that all data collected in this application (such as sample data pairs, target text, and various model-related data) were collected with the consent and authorization of the data subject (such as users, institutions, or enterprises), and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions.

[0092] Please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture for generating images from text, provided in an embodiment of this application. Figure 1 As shown, this network architecture may include a server 200 and a terminal device cluster. The terminal device cluster may include one or more terminal devices; the number of terminal devices is not limited here. Figure 1 As shown, multiple terminal devices can specifically include terminal device 1, terminal device 2, terminal device 3, ..., terminal device n, where n is a positive integer. For example... Figure 1 As shown, terminal device 1, terminal device 2, terminal device 3, ..., terminal device n can all connect to server 200 via the network, so that each terminal device can interact with server 200 through the network connection.

[0093] like Figure 1 The server 200 shown can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal device can be a smartphone, tablet, laptop, desktop computer, smart TV, in-vehicle terminal, smart home device, or other smart terminal. The following description uses the communication between terminal device 1 and server 200 as an example to illustrate the specific implementation of this application.

[0094] In this context, terminal device 1 can be a user terminal, and terminal device 1 may contain a business client. This business client is a client capable of requesting the generation of an image that matches the input text. This client can be software, a mini-program (a program that does not require installation), a webpage, etc. Server 200 can be the backend server to which this business client belongs.

[0095] Therefore, a user can input a string of text (referred to as target text) into the business client of terminal device 1 to request the generation of an image that matches (i.e., conforms to) the text content of the target text. The business client in terminal device 1 can submit the target text to server 200, which can generate an image (referred to as target image) that matches the text content of the target text using a trained noise reduction model. Server 200 can then return the generated target image to the business client in terminal device 1, allowing the business client to output the target image for the user to view and use. This specific process can also be found in the relevant descriptions in the following embodiments.

[0096] Please see also Figure 2 , Figure 2 This is a schematic diagram illustrating a scenario for training a noise reduction model, as provided in an embodiment of this application. Figure 2The content shown describes the process of the server 200 training the denoising model. Specifically, the server 200 can obtain sample data pairs for training the denoising model. The sample data pairs can include sample text and sample images that match the text content of the sample text. That is, the text content of the sample text and the image content of the sample image are matched. For example, the sample image may depict the things or scenes described by the sample text.

[0097] Server 200 can first perform denoising processing on the sample image to generate a first noise map to be denoised. Then, server 200 can call the denoising model (the denoising model that needs to be trained) to perform N denoising processing on the first noise map, thereby generating denoising features corresponding to each denoising processing step, for a total of N denoising features (including denoising feature h1 to denoising feature hN here), where N is a positive integer.

[0098] Server 200 can select M denoising features (including denoising features r1 to rM) from the N denoising features to implement loss feedback for the denoising model, where M is a positive integer less than N. Server 200 can generate M feedback losses (including feedback loss 1 to feedback loss M) for the denoising model using these M denoising features. One of the M denoising features can be used to generate a corresponding type of feedback loss. Server 200 can use these M feedback losses to perform feedback training on the denoising model, thereby obtaining a trained denoising model. This specific process can also be found in the corresponding descriptions in the following embodiments.

[0099] Using the method provided in this application, the denoising model can be trained with feedback on different types of effects by using the denoising features obtained in different denoising steps, thereby training a denoising model with excellent performance. The denoising model trained in this way can be used to accurately generate images that match the text content of the input text.

[0100] Please see Figure 3 , Figure 3 This is a flowchart illustrating a model processing method provided in an embodiment of this application. The executing entity in this embodiment can be a model processing device (hereinafter referred to as a processing device). This model processing device can be a single computer device or a cluster of multiple computer devices. The computer device can be a server, a terminal device, or other devices; there are no limitations on this. Figure 3 As shown, the method may include:

[0101] Step S101: Obtain sample data pairs, which include sample text and sample images that match the text content of the sample text.

[0102] Specifically, the processing device can acquire sample data pairs, which are used to train the denoising model. This application can have a large number of sample data pairs, the specific number of which can be determined according to the actual application scenario. This application can use these large number of sample data pairs to achieve batch-by-batch training of the denoising model. For example, the denoising model can be trained for 50 rounds of iterations. Each round of iterations can be trained using multiple batches of sample data pairs, and a batch of sample data pairs can include multiple sample data pairs. Furthermore, since the principle of processing each sample data pair is the same during training, it can be understood that the following data processing in this application is collectively referred to as a sample data pair, without emphasizing the number of sample data pairs. Moreover, only the sample images and sample texts within the same sample data pair are associated; the sample images and sample texts in different sample data pairs are processed independently.

[0103] The sample data pair may include sample text and a sample image that matches the text content of the sample text. That is, the sample data pair may be a text-image pair. The sample text and sample image in different sample data pairs may be different, but the sample image in a sample data pair should be an image that matches (i.e., is consistent with) the text content of the sample text in the sample data pair.

[0104] It is understandable that matching the sample image with the text content of the sample text means that the image content presented by the sample image can be the text content described by the sample text. In other words, the sample text can be the text used to describe the image content of the sample image.

[0105] For example, a sample data pair may include sample text w1 and sample image t1. Sample text w1 could be the text "Guilin landscape," and sample image t1 could be an image depicting a scene of Guilin's landscape. Similarly, a sample data pair may include sample text w2 and sample image t2. Sample text w2 could be the text "A family eating New Year's Eve dinner," and sample image t2 could be an image depicting a family eating New Year's Eve dinner at home, decorated with festive New Year's decorations. And so on.

[0106] Optionally, the above sample data pairs can be obtained through various collection methods, and this application does not limit the collection method of the sample data pairs.

[0107] Step S102: Add noise to the sample image to generate a first noise map, and call the denoising model to perform N-step denoising on the first noise map based on the sample text, generating denoising features corresponding to each denoising step, where N is a positive integer.

[0108] Specifically, the processing device can add noise to the sample images in the sample data pair to generate a first noise map to be denoised, and this process can be described as follows.

[0109] The processing device can invoke an image encoder to perform feature encoding on the sample image to generate image-encoded features of the sample image. These image-encoded features can be feature maps (such as features in matrix form). The image encoder can be a pre-trained model that can be used for feature encoding of images. The process of the image encoder performing feature encoding on the sample image is the process of the image encoder learning and extracting features from the sample image. The image-encoded features obtained by encoding the sample image can be understood as the features obtained by mapping the sample image to the latent feature space of the image encoder. Optionally, the image encoder can be a VAE (an autoencoder).

[0110] Furthermore, the processing device can add noise to the image coding features of the sample image to generate an initial noise map. Optionally, the processing device can call a denoising model to randomly generate a sample noise map. For example, the sample noise map can be a Gaussian noise map randomly generated by the denoising model. The dimension of the Gaussian noise map can be the same as the dimension of the image coding features of the sample image. Therefore, the processing device can sum the sample noise map and the image coding features of the sample image (i.e., add them together) to obtain the initial noise map. For example, the processing device can call the denoising model to first randomly generate a seed for generating Gaussian noise. This seed can be a random number, so that the denoising model can generate the corresponding Gaussian noise map using this seed.

[0111] The summation of the image coding features of the sample noise map and the sample image can be achieved by adding the element values ​​at the same position (such as the same row and column position) in the sample noise map and the image coding features.

[0112] Furthermore, the processing device can also map the initial noise map to the latent feature space of the denoising model to obtain the first noise map. Specifically, the processing device can invoke the denoising model to perform feature mapping processing on the initial noise map to generate a first noise map in the latent feature space of the denoising model.

[0113] Optionally, the denoising model in this application can be a diffusion model, such as the stable-diffusion model (an open-source diffusion model, which can be abbreviated as SD model). This diffusion model can be used to perform diffusion processing. Therefore, calling the denoising model to perform feature mapping processing on the initial noise map can be called to perform diffusion processing (i.e., diffusion processing) on ​​the initial noise map. This diffusion processing can be used to map the initial noise map to the latent feature space of the denoising model, so that the subsequent denoising model can more adaptively process the first noise map (such as denoising processing).

[0114] The processing device can also call the noise reduction model to perform N-step noise reduction processing (also known as N-times noise reduction processing) on ​​the first noise map through sample text. The N-step noise reduction processing can be performed iteratively. Each subsequent noise reduction processing can be a further noise reduction processing based on the previous noise reduction processing. For example, the second noise reduction processing can be performed on the result obtained from the first noise reduction processing, the third noise reduction processing can be performed on the result obtained from the second noise reduction processing, and so on.

[0115] In other words, the denoising model can perform several denoising processes (also known as noise reduction) on the first noisy image under the supervision (i.e., influence) of the sample text (such as under the influence of cross-attention of the sample text), and finally obtain the features used to generate an image that matches the text content of the sample text. Here, N is a positive integer, such as N equal to 50. The specific value of N can be set according to the actual application scenario, and this application does not impose any restrictions on it.

[0116] Optionally, the processing device can acquire a text encoder, which can be a trained model capable of feature encoding text. The processing device can invoke the text encoder to perform feature encoding on the sample text to generate text-encoded features of the sample text. The processing device can then input these text-encoded features into a denoising model, allowing the denoising model to sequentially perform N-step denoising on the first noise map using these text-encoded features. This achieves denoising of the first noise map using the sample text.

[0117] Optionally, the aforementioned text encoder can be a network structure used for text processing in a text-image processing model. This text-image processing model can be a Clip model (a multimodal neural network) that aims to achieve cross-modal understanding between text and images by learning the contrastive relationships between them. The core idea of ​​the Clip model is to embed text and images into a common semantic space (i.e., feature space), so that the representations of related text and images in this semantic space are close to each other, while the representations of unrelated text and images are far apart.

[0118] Optionally, the denoising model of this application may include a U-Net (a convolutional neural network) module. The aforementioned N-step denoising processes can all be performed by repeatedly calling this U-Net module. The key focus of this application is the training of this U-Net module, through which features can be obtained to finally generate an image that matches the text content of the input text (such as sample text). The aforementioned N-step denoising processes can be N forward computations of denoising U-Net (the denoising process of U-Net).

[0119] After the processing device calls the denoising model to perform N denoising steps on the first noise map using sample text, it can generate denoising features corresponding to each denoising step. These denoising features can also be feature maps. In other words, after each denoising step of the first noise map is performed using the denoising model with sample text, a corresponding denoising feature is generated. Each denoising step can correspond to one denoising feature, which is obtained by the denoising model after performing that denoising step on the first noise map using sample text. Therefore, after performing N denoising steps on the first noise map using sample text, a total of N denoising features corresponding to the N denoising steps can be obtained.

[0120] Specifically, how the denoising model denoises the first noise map using sample text is the learning process required for the denoising model, which is the process of training the denoising model.

[0121] In one feasible implementation, the U-Net module described above may be composed of one or more QKV structures (a structure of an attention mechanism), where Q can be understood as the information to be queried, K can be understood as the vector being queried, and V can be used to represent the value obtained from the query. In this application, the denoising processing of the first noise map may be implemented through each QKV structure in the U-Net module.

[0122] A QKV structure can contain multiple stacked Res Blocks (convolutional blocks with skip connections) and Spatial Transformer structures (a type of spatial transformation structure belonging to convolutional networks). Typically, a QKV structure can contain two Spatial Transformer structures. One Spatial Transformer structure represents one QKV process of the QKV structure (which can be understood as an attention mechanism process). In the first QKV process of a QKV structure, Q, K, and V can all be inputs to the QKV structure. In the second QKV process, Q can be the output of the previous network layer, while K and V in the second QKV process can both be the text encoding features of the sample text. The second QKV process can control (i.e., constrain) the denoising effect on the first noisy map through K and V (i.e., the text encoding features of the sample text). The input to the first QKV structure can be the first noisy map, the input to the second QKV structure can be the output of the first QKV structure, and so on.

[0123] Please see Figure 4 , Figure 4 This is a schematic diagram of a neural network structure provided in an embodiment of this application. For example... Figure 4 As shown, a complete QKV structure in U-Net can include two Res Blocks (residual modules, namely ResBlock1 and ResBlock2), two Spatial Transformers (learning modules, namely Spatial Transformer1 and Spatial Transformer2), and one Downsample module. In Spatial Transformer1, Q, K, and V can be the same and can all be inputs to this QKV structure. In Spatial Transformer2, Q can be the output of Res Block2, and K and V can both be text embeddings, which can be the text encoding features of the sample text mentioned above.

[0124] In one implementation, the denoising model can be obtained by fine-tuning an initial denoising model using sample data. By using this sample data to fine-tune the initial denoising model, the model can first acquire basic generative capabilities (such as feature generation capabilities, which are also basic denoising capabilities). Based on this, further optimization adjustments with loss feedback can be made to the denoising model under different denoising steps, improving the training effect and ultimately resulting in a superior denoising model. The specific process of fine-tuning the initial denoising model to obtain the denoising model can be found below. Figure 10 The description in the corresponding embodiment.

[0125] Step S103: From the N noise reduction features corresponding to the N-step noise reduction process, obtain M noise reduction features for loss feedback of the noise reduction model, where M is a positive integer and M is less than N.

[0126] Specifically, the processing device can extract M denoising features from the N denoising features corresponding to the N-step denoising process generated above, which are used for loss feedback (also known as information feedback) of the denoising model. M is a positive integer, and M is less than N. Alternatively, in some special application scenarios, M can be set to be equal to N, depending on the actual application scenario. The M-step denoising process corresponding to these M denoising features can be called the M-step feedback denoising process. In practical applications, the M-step feedback denoising process can be selected first from the N-step feedback denoising process, and then the denoising features corresponding to the M-step feedback denoising process can be used as the M denoising features obtained for loss feedback of the denoising model.

[0127] In general, this application can select M denoising features from the aforementioned N denoising features, and use these M denoising features to implement specific types of loss feedback on the denoising model, thereby achieving model training for specific effects of the denoising model, as described below. Specifically, the specific types of loss feedback on the denoising model using these M denoising features means using these M denoising features to perform reinforcement learning on the denoising model in multiple optimization directions (i.e., multiple optimization metrics). This allows the denoising model to obtain more supervision information, thus enabling it to be trained and learned towards high-quality image generation without incurring higher costs in collecting more high-quality sample data pairs, achieving the effect of reinforcement learning for the denoising model.

[0128] Optionally, the processing device can obtain the set first step number i and second step number j, where i and j are both positive integers, and i and j are less than or equal to N. The first step number i and the second step number j are both step numbers in the above N-step noise reduction process, that is, i corresponds to the i-th noise reduction step in the N-step noise reduction process, and j corresponds to the j-th noise reduction step in the N-step noise reduction process.

[0129] Optionally, this application may have two types of feedback loss: one is the image-text matching type (also known as the image-text consistency type, i.e. the consistency between text content and image content), and the other is the image aesthetics type feedback loss. Both types of feedback loss are extremely important in the field of generating images from text.

[0130] Therefore, the first step number i mentioned above can be the number of steps used to generate the feedback loss corresponding to the image-text matching type, and the second step number j mentioned above can be the number of steps used to generate the feedback loss corresponding to the image aesthetic type.

[0131] It should be noted that in the data processing of the denoising model, the earlier the steps, the more they affect the layout of the final data (such as the image generated by the denoising features); the middle steps, the more they affect the content of the final data; and the later steps, the more they affect the aesthetics of the final data. This is an important characteristic of the model's data processing.

[0132] Therefore, this application can determine the first step number i through an intermediate step in the N-step noise reduction process, and can determine the second step number j through a later step in the N-step noise reduction process. Thus, i can be less than j. For example, if N equals 50, this application can randomly determine the first step number i within the range of the first step number, and can randomly determine the second step number j within the range of the second step number. The number of steps in the second step range is greater than the number of steps in the first step range. For example, the range of the first step number can be 30 to 40, and the range of the second step number can be 41 to 50. The determined first step number i can be 33, and the determined second step number j can be 45.

[0133] Optionally, the range of the first step number and the range of the second step number can be preset. Based on this, the first step number i and the second step number j can also be predetermined using these ranges. In this case, the first step number i and the second step number j can be fixed and the same for different training rounds of the denoising model. Alternatively, the first step number i and the second step number j can be randomly determined in real-time using these ranges for each training round of the denoising model. In this case, the first step number i and the second step number j can be different for different training rounds of the denoising model. The specific method for determining the first step number i and the second step number j can be determined according to the actual application scenario, and this application does not impose any restrictions on this.

[0134] Therefore, the processing device can select the noise reduction features corresponding to the i-th and j-th steps from the above N noise reduction features as the M noise reduction features, where M can be equal to 2. The noise reduction feature corresponding to the i-th step is used to generate the feedback loss of the image-text matching type, and the noise reduction feature corresponding to the j-th step is used to generate the feedback loss of the image aesthetics type.

[0135] Optionally, it should be noted that the above-mentioned image-text matching type feedback loss and image aesthetics type feedback loss in this application are only illustrative examples. In actual application scenarios, more than 2 denoising features can be selected from the above N denoising features to perform different types of loss feedback on the denoising model. The specific number of denoising features to be selected and the type of loss feedback to be performed on the denoising model can be flexibly selected according to the actual application scenario, and this application does not impose any restrictions on this.

[0136] Step S104: Perform loss feedback processing based on the M denoising features to generate a feedback loss for the denoising model. Each of the M denoising features corresponds to a type of feedback loss.

[0137] Specifically, the processing device can perform loss feedback processing using the M denoising features obtained above to generate a feedback loss for the denoising model. Each of the M denoising features can generate a corresponding type of feedback loss for the denoising model; that is, one denoising feature among the M features can generate a corresponding type of feedback loss for the denoising model, and different denoising features among the M features can be used to generate different types of feedback losses for the denoising model.

[0138] Therefore, after loss feedback processing using M denoising features, M feedback losses can be generated for the denoising model. The specific process of generating these M feedback losses based on the M denoising features can be found below. Figure 6 and Figure 8 For the specific process of generating the feedback loss of the image-text matching type corresponding to the relevant descriptions in the embodiments, please refer to the following. Figure 6 For the description in the corresponding embodiments, and the specific process of generating feedback loss of image aesthetics type, please refer to the following: Figure 8 The description in the corresponding embodiment.

[0139] Step S105: Based on the generated M feedback losses, correct the model parameters of the denoising model to obtain the trained denoising model. The trained denoising model is used to generate an image that matches the text content of the input text.

[0140] Specifically, the processing device can use the M feedback losses generated above to modify the model parameters of the noise reduction model in multiple ways, as described below.

[0141] The processing device can also obtain the image generation loss of the denoising model. This image generation loss can be the overall loss of image generation through the denoising model. This image generation loss can be used to reflect the overall deviation of the denoising model when generating images. The larger the image generation loss, the larger the deviation of the denoising model when generating images. Conversely, the smaller the image generation loss, the smaller the deviation of the denoising model when generating images.

[0142] Specifically, the process of obtaining the image generation loss may include: the processing device may call the image decoder to perform feature decoding processing on the denoising features corresponding to the Nth denoising process (i.e. the last denoising process) in the above N-step denoising process, and generate the corresponding image, which may be referred to as the third predicted image. The third predicted image is the image finally generated by the denoising features corresponding to the Nth denoising process.

[0143] The image decoder can be a pre-trained model that generates corresponding images from denoised features generated by the denoising model. The image decoder can use the decoding principle corresponding to the VAE encoder to decode the denoised features. That is, this application can use the decoding principle of VAE to decode the denoised features.

[0144] The processing device can generate the image generation loss of the denoising model based on the difference between the obtained third predicted image and the sample images in the sample data pair. For example, this image generation loss can be the MSE (mean squared error loss) between the third predicted image and the sample images. This MSE loss reflects the difference between the third predicted image and the sample images; the larger the MSE loss, the greater the difference between the third predicted image and the sample images, and vice versa. The MSE loss between the third predicted image and the sample images (which can be denoted as MSE1) can be expressed by the following formula:

[0145]

[0146] In this model, the dimension of the third predicted image is the same as that of the sample image. The number of pixels in both the sample image and the third predicted image can be L, where L is a positive integer. The specific value of L can be determined according to the actual application scenario. k This represents the pixel value of the k-th pixel in the sample image. This represents the pixel value of the k-th pixel in the third predicted image, where k is a positive integer less than or equal to L, and k can take values ​​from 1 to L.

[0147] Therefore, the processing device can sum the generated image generation loss with each of the M feedback losses to generate M total feedback losses. Summing the image generation loss with one of the M feedback losses (i.e., adding them together) generates a total feedback loss.

[0148] In one feasible implementation, the method of summing the image generation loss and each feedback loss to generate M total feedback losses can include:

[0149] The processing device can obtain the weight coefficients corresponding to the image generation loss. These weight coefficients can be called the first weight coefficients. The first weight coefficients can be used to reflect the proportion of the image generation loss in the process of correcting the model parameters of the denoising model (which can be understood as the degree of importance or influence). The larger the first weight coefficient, the greater the proportion of the image generation loss in the process of correcting the model parameters of the denoising model. Conversely, the smaller the first weight coefficient, the smaller the proportion of the image generation loss in the process of correcting the model parameters of the denoising model.

[0150] The processing device can also obtain the weight coefficient corresponding to each of the M feedback losses mentioned above. This weight coefficient can be referred to as the second weight coefficient. Optionally, the second weight coefficients corresponding to different feedback losses can be the same or different, depending on the actual application scenario. Similarly, the second weight coefficients corresponding to each feedback loss can also be used to reflect the proportion of each feedback loss in the process of correcting the model parameters of the denoising model.

[0151] The first weight coefficient corresponding to the image generation loss and the second weight coefficient corresponding to each feedback loss can be flexibly set according to the actual application scenario. Higher weight coefficients can be assigned to losses that are more important. For example, the first weight coefficient can be set to 1, and the second weight coefficients corresponding to the M feedback losses can all be set to 0.01.

[0152] Therefore, the processing device can use the first weighting coefficient obtained above and the second weighting coefficient corresponding to each feedback loss to sum the image generation loss with each feedback loss to generate the above M total feedback losses. For example, if the first weighting coefficient is q1, the image generation loss can be represented as s1, and the second weighting coefficient corresponding to a feedback loss is q2, the feedback loss can be represented as s2. Then the processing device can calculate the formula s1*q1+s2*q2 to generate a total feedback loss corresponding to that feedback loss. The "*" in this formula can represent multiplication.

[0153] The processing device can send the M total feedback losses generated above back to the denoising model, so as to correct the model parameters of the denoising model in turn through the M total feedback losses. The subsequent total feedback loss can be based on the result of the previous total feedback loss correcting the model parameters of the denoising model, and further iteratively correct the model parameters of the denoising model.

[0154] The model parameters of the denoising model can be corrected sequentially using the total feedback loss, following an ascending order of the step number (e.g., step i and step j mentioned above). For example, the model parameters can be corrected first using the total feedback loss corresponding to the image-text matching type. After correction, the model parameters can be further corrected using the total feedback loss corresponding to the image aesthetics type. The goal is to correct the model parameters so that each total feedback loss approaches its minimum value (e.g., approaches 0). In other words, when correcting the model parameters of the denoising model using a total feedback loss, the model parameters can be adjusted to minimize that total feedback loss.

[0155] Optionally, the model parameters of the denoising model can be corrected using the total loss from the aforementioned feedback methods through the SGD (Stochastic Gradient Descent) method. When initializing and reinforcing the denoising model, a learning rate of 0.0004 can be used, and then adjusted to 0.1 times the original rate after every 5 iterations, thus performing multiple rounds (e.g., 50 rounds) of reinforcement training. Alternatively, other learning rates can be used during training; this application does not impose any restrictions on this.

[0156] The processing device can be trained in multiple batches and rounds according to the principle described above (i.e., the model parameters of the denoising model are corrected in multiple batches and rounds). When the training is completed (such as when the number of iterative training rounds of the denoising model reaches the specified number of training rounds, or when the model parameters of the denoising model are trained to a convergent state), the trained denoising model can be obtained.

[0157] Alternatively, in a feasible implementation, the denoising model can be iteratively trained multiple times according to the above principles. Each feedback training can reinforce the denoising model using a type of feedback loss. For example, the above process can describe any feedback training of the denoising model, where the M denoising features obtained in this process may contain only one denoising feature, which can be used to perform a single feedback training of the denoising model. Subsequent feedback training can be further iterative training based on the previous feedback training.

[0158] For example, if the denoising model needs to be trained in two iterations using the feedback loss of image-text matching and image aesthetics, then only the denoising features corresponding to the denoising process of step i generated in the first feedback training process can be used to achieve the first feedback training of the denoising model. After the first feedback training is completed, the denoising features corresponding to the denoising process of step j generated in the second feedback training process can be used to achieve the second feedback training of the denoising model, and finally the trained denoising model is obtained.

[0159] Alternatively, in another feasible implementation, if a total of F types of feedback loss are required to perform G iterations of feedback training on the denoising model, where F and G are both positive integers and F is greater than or equal to G, then each iteration can use a subset of the F types of feedback loss (such as one or more types) to perform G iterations of feedback training on the denoising model. That is, a single feedback training iteration can use a subset of the F types of feedback loss to train the denoising model, and the types of feedback loss used in different iterations of feedback training can be different. In this case, the above process can also describe any iteration of feedback training on the denoising model, and the aforementioned M denoising features can be the M types of feedback loss selected from the F types of feedback loss for any iteration of feedback training on the denoising model.

[0160] The specific methods for implementing feedback training of the denoising model using various types of feedback loss can be selected according to the actual application scenario. It should be noted that any scheme that applies different types of loss feedback to the model at different data processing steps (i.e., stages) is within the scope of protection of this application.

[0161] After the trained denoising model is obtained, the processing device can use the trained denoising model to generate images, as described below.

[0162] The processing device can receive an image generation request submitted by a business client. This request can be generated by the business client and may carry target text. This target text may be entered (e.g., input) by the owner of the business client (e.g., the user to whom the business client belongs) within the business client. The image generation request can be used to request the generation of an image that matches the text content of the target text. For example, the target text may be a string of text descriptions entered by the user.

[0163] Therefore, the processing device can input the target text into a trained denoising model, which will then generate an image that matches the text content of the target text. This generated image can be called the target image. The processing device can then return the generated target image to the business client, allowing the client to output the target image for the user to view. This achieves the goal of quickly and accurately generating an image depicting the content described by a string of text provided by the user.

[0164] The specific process by which the processing device calls the trained noise reduction model to generate a target image that matches the text content of the target text may include:

[0165] The processing device can call a trained denoising model to first randomly generate a noise map, which can be referred to as the third noise map. The processing device can then use the trained denoising model to denoise this third noise map using the target text (or perform N-step denoising processes according to the same principle described above) to generate target denoising features. These target denoising features can be the denoising features corresponding to the last denoising step. In other words, the trained denoising model can denoise the third noise map under the supervision (influence) of the target text to obtain denoising features used to generate an image that matches the text content of the target text, i.e., the target denoising features.

[0166] The principle of performing N-step noise reduction on the third noise map using the target text is the same as the principle of performing N-step noise reduction on the first noise map using the sample text.

[0167] The processing device can call the aforementioned image decoder to perform feature decoding processing on the target denoising features, thereby generating the aforementioned target image. This target image is the image ultimately generated by the trained denoising model that matches the text content of the target text provided by the user.

[0168] Alternatively, there can be multiple target denoising features. For example, these multiple target denoising features can be denoising features corresponding to later denoising steps in the N-step denoising process of the third noise image. For instance, there can be three target denoising features, which may include the denoising feature corresponding to the (N-2)th denoising step, the (N-1)th denoising step, and the Nth denoising step. Thus, the processing device can call the image decoder to perform feature decoding processing on each of these multiple target denoising features to generate multiple target images. One target denoising feature can be used to decode and generate one target image.

[0169] The processing device can return all the target images to the business client, and the business client can simultaneously output the multiple target images for the user to select and use.

[0170] In one feasible application scenario, the text in this application can also be the name of an emoticon, and the image generated from the text in this application can be an emoticon that matches the text content. For example, if the target text is "happy," then the generated target image can be a smiling emoticon; or, if the target text is "great," then the generated target image can be an emoticon showing a "great" gesture. However, it should be noted that this application can be applied to any application scenario that requires generating a matching image from text.

[0171] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating the effect of a client interface for a business client provided in an embodiment of this application. For example... Figure 5 As shown, the client interface can be used to request the generation of an image that matches the text content of the input text. It supports inputting text (such as target text) in box 100a and clicking the confirmation control below box 100a. The business client can then submit the target text to the processing device to request the processing device to generate a target image that matches the text content of the target text.

[0172] The processing device can return the generated target images to the business client, so that the business client can display the received target images in box 101a of the client interface. Each target image can be displayed with a selection control (such as the circular control here) below it, allowing the user to select the target images displayed in box 101a through the selection control of each target image. The target images selected by the user can also be displayed in the historical results display area 102a of the client interface. Here, the historical results display area 102a can also display the emoticons that the user previously selected, such as the starry-eyed emoticon, the happy emoticon, and the injured emoticon.

[0173] The denoising model trained using the method described in this application can generate images that closely match the description of the input text and have high aesthetic evaluation (such as no distortion, no missing objects, clear images, and beautiful layout), thereby meeting the user's needs and improving the user experience.

[0174] This application can acquire sample data pairs, including sample text and sample images matching the text content of the sample text; it can add noise to the sample images to generate a first noise map, and call a denoising model to perform N-step denoising on the first noise map based on the sample text, generating denoising features corresponding to each denoising step, where N is a positive integer; thus, from the N denoising features corresponding to the N-step denoising, M denoising features for loss feedback to the denoising model can be obtained, where M is a positive integer and M is less than N; this application can also perform loss feedback processing based on the M denoising features to generate feedback loss for the denoising model, where each of the M denoising features corresponds to a type of feedback loss; the model parameters of the denoising model are corrected based on the generated M feedback losses to obtain a trained denoising model, which is used to generate an image matching the text content of the input text. Therefore, the method proposed in this application can select denoising features (such as the selected M denoising features) corresponding to some steps of denoising processing in multiple steps of denoising processing of a noisy map (such as the first noisy map) (such as N steps of denoising processing) to generate different types of feedback losses for the denoising model. Thus, the model parameters of the denoising model can be more comprehensively and accurately corrected through the generated various types of feedback losses, and finally a well-trained denoising model can be obtained. The well-trained denoising model can then generate an image that matches the text content of the input text and has excellent effect.

[0175] Please see Figure 6 , Figure 6 This is a schematic diagram of a process for generating a feedback loss of image-text matching type according to an embodiment of this application. For example... Figure 6 As shown, the process may include:

[0176] Step S201: Call the image decoder to perform feature decoding processing on the denoising features corresponding to the denoising process in step i, and generate the first predicted image.

[0177] Specifically, the processing device can call the image decoder to perform feature decoding processing on the denoising features corresponding to the i-th step of the denoising process to generate the corresponding image. The generated image can be called the first predicted image, which is the image generated by decoding the denoising features corresponding to the i-th step of the denoising process.

[0178] Step S202: Perform image-text matching processing on the first predicted image and the sample text to obtain the image-text matching degree between the first predicted image and the sample text.

[0179] Specifically, the loss feedback processing based on the M noise reduction features mentioned above may include image-text matching processing, which may be a matching process between the first predicted image generated by the above decoding and the sample text, as described below.

[0180] The processing device can acquire an image processing model, which can be a pre-trained model that can map input images and text to the same latent feature space. For example, the image processing model can be a Clip model (a type of multimodal neural network).

[0181] Therefore, the processing device can invoke the image-text processing model to perform feature transformation processing on the first predicted image to generate transformed image features of the first predicted image. The processing device can also invoke the image-text processing model to perform feature transformation processing on sample text to generate transformed text features of the sample text. The transformed image features and the transformed text features can then reside in the same feature space (i.e., latent feature space) of the image-text processing model.

[0182] The processing device can calculate the feature matching degree between the transformed image features and the transformed text features, and the feature matching degree can be used to reflect the degree of matching between the image content of the first predicted image and the text content of the sample text.

[0183] For example, both the transformed image features and the transformed text features can be feature vectors. The processing device can calculate the cosine similarity between the transformed image features and the transformed text features as the feature matching degree between them. Therefore, the processing device can use the feature matching degree between the transformed image features and the transformed text features as the image-text matching degree between the first predicted image and the sample text. The larger the image-text matching degree, the more closely the image content of the first predicted image matches the text content of the sample text (i.e., the more consistent they are); conversely, the smaller the image-text matching degree, the less closely the image content of the first predicted image matches the text content of the sample text (i.e., the more inconsistent they are).

[0184] Based on the above description, it can be understood that the above image-text matching process can include feature transformation processing of the first predicted image, feature transformation processing of the sample text, and calculation of the feature matching degree between the transformed image features and the transformed text features.

[0185] Step S203: Generate the image-text matching loss of the denoising model based on the image-text matching degree; the image-text matching loss belongs to the image-text matching type of loss.

[0186] Specifically, the processing device can generate the image-text matching loss of the denoising model based on the image-text matching degree obtained above. This image-text matching loss can be classified as an image-text matching type loss. This loss reflects the deviation in matching between the image generated by the denoising model and the sample text. A larger image-text matching loss indicates a greater deviation in matching between the image generated by the denoising model and the sample text; conversely, a smaller image-text matching loss indicates a smaller deviation. The image-text matching loss and the image matching degree can be negatively correlated; that is, a higher image matching degree results in a smaller image-text matching loss, and vice versa.

[0187] For example, the image-text matching loss can be 1 minus the image matching degree. For instance, the image-text matching loss can be represented as ps, and the image matching degree can be represented as cos, then ps can be equal to 1-cos.

[0188] Step S204: Based on the image-text matching loss, generate the feedback loss for the denoising model in the i-th step of denoising processing.

[0189] Specifically, the dimension of the denoising feature corresponding to the i-th denoising step can be the same as the dimension of the first noise map. The processing device can calculate the difference between the denoising feature corresponding to the i-th denoising step and the first noise map (i.e., the input of U-Net) as the predicted noise map corresponding to the i-th denoising step. The predicted noise map is the noise in the first noise map predicted after the i-th denoising step is performed on the first noise map.

[0190] Furthermore, since the first noise map is obtained by adding sample noise to the image coding features of the sample image, the processing device can generate the noise prediction loss corresponding to the i-th step of denoising by the difference between the predicted noise map corresponding to the i-th step of denoising and the original sample noise map added to the first noise map. This noise prediction loss can be referred to as the first noise prediction loss. For example, the first noise prediction loss can also be the MSE loss (mean squared error loss) between the predicted noise map corresponding to the i-th step of denoising and the sample noise map. This mean squared error loss can be used to reflect the difference between the predicted noise map corresponding to the i-th step of denoising and the sample noise map.

[0191] Therefore, the processing device can sum (i.e. add) the first noise prediction loss and the generated image-text matching loss to generate the feedback loss for the denoising model in the i-th step of denoising processing. Since the image-text matching loss is introduced into the feedback loss, the feedback loss belongs to the image-text matching type of feedback loss. It can be understood that the feedback loss is obtained by introducing the first noise prediction loss on the basis of the image-text matching loss.

[0192] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating a scenario for generating a feedback loss of the text-image matching type, as provided in an embodiment of this application. For example... Figure 7 As shown, the processing device can input the denoising features corresponding to the i-th denoising step into the image decoder, so that the image decoder can perform feature decoding processing on the denoising features corresponding to the i-th denoising step to generate a first predicted image. The processing device can perform image-text matching processing between the first predicted image and the sample text to obtain the image-text matching degree between the first predicted image and the sample text. The processing device can generate the image-text matching loss of the denoising model through the image-text matching degree.

[0193] The processing device can also generate the first noise prediction loss for the denoising model in the i-th step of denoising processing using the denoising features corresponding to the denoising process in the i-th step, the first noise map after denoising processing, and the sample noise map added to the sample image. The processing device can sum the generated image-text matching loss and the first noise prediction loss to generate the image-text matching type feedback loss for the denoising model in the i-th step of denoising processing.

[0194] Through the above process, the image-text matching type feedback loss of the i-th step of denoising is generated for the denoising model. Through this feedback loss, the image-text matching type loss feedback of the i-th step of denoising can be realized, so as to train the denoising model to generate an image that is more closely matched with the text content of the input text.

[0195] Please see Figure 8 , Figure 8 This is a schematic diagram of a feedback loss method for generating image aesthetics, provided in an embodiment of this application. Figure 8 As shown, the process may include:

[0196] Step S301: Call the image decoder to perform feature decoding processing on the denoising features corresponding to the denoising process in step j, and generate the second predicted image.

[0197] Specifically, the processing device can call the image decoder to perform feature decoding on the noise reduction features corresponding to the j-th step noise reduction process to generate the corresponding image. The generated image can be called the second predicted image, which is the image generated by decoding the noise reduction features corresponding to the j-th step noise reduction process.

[0198] Step S302: Call the aesthetic evaluation model to evaluate the aesthetics of the second predicted image, generate an aesthetic score for the second predicted image, and generate the aesthetic loss of the denoising model based on the aesthetic score; the aesthetic loss belongs to the loss of image aesthetics type.

[0199] Specifically, the processing device can acquire an aesthetic evaluation model, which can be a pre-trained model used to evaluate the aesthetics of an input image. For example, this aesthetic evaluation model could be LAION-AESTHETICS (an open-source aesthetic evaluation model for assessing images).

[0200] The loss feedback processing based on the M noise reduction features described above can include image aesthetics evaluation. Therefore, the processing device can call the aesthetics evaluation model to evaluate the aesthetics of the second predicted image, generating an aesthetics score for the second predicted image. The more aesthetically pleasing the second predicted image, the higher its aesthetics score; conversely, the less aesthetically pleasing the second predicted image, the lower its aesthetics score. Specifically, the criteria for determining whether an image is aesthetically pleasing or not is learned by the aesthetics evaluation model during training.

[0201] Furthermore, the processing device can also generate an aesthetic loss for the denoising model based on the aesthetic score. This aesthetic loss falls under the category of image aesthetics loss. The aesthetic loss reflects the deviation in aesthetics of the image generated by the denoising model; a larger aesthetic loss indicates a less aesthetically pleasing image, and vice versa. The aesthetic loss and the aesthetic score can be negatively correlated; a higher aesthetic score results in a smaller aesthetic loss, and vice versa.

[0202] For example, the aesthetic loss can be obtained by operating on the aesthetic score using an activation function. For instance, the activation function could be a sigmoid function, and the aesthetic score could be represented as pf; therefore, the aesthetic loss could be equal to sigmoid(pf).

[0203] Step S303: Based on the aesthetic loss, generate the feedback loss of the noise reduction process in step j for the noise reduction model.

[0204] Specifically, the dimension of the denoising feature corresponding to the j-th denoising step can be the same as the dimension of the first noise map. The processing device can calculate the difference between the denoising feature corresponding to the j-th denoising step and the first noise map (i.e., the input of U-Net) as the predicted noise map corresponding to the j-th denoising step. The predicted noise map is the noise in the first noise map predicted after the j-th denoising step is performed on the first noise map.

[0205] Furthermore, since the first noise map is obtained by adding sample noise to the image coding features of the sample image, the processing device can generate the noise prediction loss corresponding to the j-th step denoising process by using the difference between the predicted noise map corresponding to the j-th step denoising process and the original sample noise map added to the first noise map. This noise prediction loss can be referred to as the second noise prediction loss. For example, the second noise prediction loss can also be the MSE loss (mean squared error loss) between the predicted noise map corresponding to the j-th step denoising process and the sample noise map. This mean squared error loss can be used to reflect the difference between the predicted noise map corresponding to the j-th step denoising process and the sample noise map.

[0206] Therefore, the processing device can sum (i.e. add) the second noise prediction loss and the aesthetic loss generated above to generate the feedback loss of the noise reduction model for the j-th step noise reduction process. Since the aesthetic loss is introduced into the feedback loss, the feedback loss belongs to the image aesthetics type of feedback loss. It can be understood that the feedback loss is obtained by introducing the second noise prediction loss on the basis of the aesthetic loss.

[0207] Please see Figure 9 , Figure 9 This is a schematic diagram illustrating a scenario of feedback loss for the aesthetic quality of generated images, as provided in an embodiment of this application. Figure 9 As shown, the processing device can input the denoising features corresponding to the j-th denoising step into the image decoder, so that the image decoder can perform feature decoding processing on the denoising features corresponding to the j-th denoising step to generate a second predicted image. The processing device can also call the aesthetic evaluation model to evaluate the aesthetics of the second predicted image, thereby obtaining the aesthetics score of the second predicted image. The processing device can use the aesthetics score to generate the aesthetics loss of the denoising model.

[0208] The processing device can also generate a second noise prediction loss for the denoising model in the j-th step of denoising, using the denoising features corresponding to the j-th step of denoising, the first noise map after denoising, and the sample noise map added to the sample image. The processing device can sum the aesthetic loss and the second noise prediction loss to generate a feedback loss of image aesthetics type for the j-th step of denoising.

[0209] Through the above process, the feedback loss of the j-th step of denoising processing on the image aesthetics type of the denoising model is generated. Through this feedback loss, the loss feedback of the j-th step of denoising processing on the image aesthetics type of the denoising model can be realized, so as to train the denoising model to generate images with higher aesthetics.

[0210] Please see Figure 10 , Figure 10 This is a schematic diagram illustrating a process for training an initial denoising model to obtain a denoising model, as provided in an embodiment of this application. Figure 10 As shown, the process may include:

[0211] Step S401: Obtain the initial noise reduction model and add noise to the sample image to generate the second noise map.

[0212] Specifically, the processing device can acquire an initial denoising model, which can be a model to be fine-tuned. The processing device can add noise to the sample images in the sample data pair to generate a second noise map. The principle of adding noise to the sample images to generate the second noise map is the same as the principle of adding noise to the sample images to generate the first noise map.

[0213] Step S402: Call the initial denoising model to denoise the second noise map based on the sample text and generate sample denoising features.

[0214] Specifically, the processing device can call the initial denoising model to denoise the second noise map by using the sample text in the sample data pair (or it can perform N-step denoising according to the same principle as above), and generate sample denoising features. These sample denoising features can be the denoising features corresponding to the last denoising step here, that is, the sample denoising features can be the denoising features generated after the Nth denoising process on the second denoised map.

[0215] The principle of performing N-step noise reduction on the second noise map using sample text is the same as the principle of performing N-step noise reduction on the first noise map using sample text.

[0216] Step S403: Call the image decoder to perform feature decoding processing on the sample denoising features and generate the sample prediction image.

[0217] Specifically, the processing device can call the aforementioned image decoder to perform feature decoding processing on the aforementioned sample denoising features in order to decode and generate the corresponding image, which can be referred to as the sample prediction image.

[0218] Step S404: Based on the difference between the predicted sample image and the sample image, correct the model parameters of the initial denoising model to obtain the denoising model.

[0219] Specifically, the processing device can correct the model parameters of the initial denoising model by analyzing the differences between the generated sample prediction image and the sample images in the sample data pair, in order to obtain the above-mentioned... Figure 3 The noise reduction model in the corresponding embodiment.

[0220] Optionally, the processing device can first generate an initial image generation loss for the initial denoising model based on the difference between the predicted image and the sample image. This loss can be referred to as the initial image generation loss of the initial denoising model, and the concept of the initial image generation loss is the same as that of the image generation loss of the denoising model described above. For example, the initial image generation loss can be the MSE loss (mean squared error loss) between the predicted image and the sample image.

[0221] The processing device can feed back the initial image generation loss to the initial denoising model, using this initial image generation loss to correct the model parameters of the initial denoising model. The goal of this correction is to adjust the model parameters of the initial denoising model so that the initial image generation loss approaches its minimum value (e.g., approaches 0). Furthermore, the processing device can perform multiple batches and multiple rounds of iterative correction (i.e., iterative training) of the model parameters of the initial denoising model according to the principles described above. When the training rounds reach the set number of training rounds or the model parameters of the initial denoising model are corrected to a convergent state, the initial denoising model trained at this point can be used as the aforementioned denoising model, and further processing of this denoising model can be performed as described above. Figure 3 The corresponding embodiment describes a multi-step, multi-feedback training process.

[0222] Optionally, the initial image generation loss can be applied using the SGD (Stochastic Gradient Descent) method to correct the model parameters of the initial denoising model. When initializing and fine-tuning the initial denoising model, a learning rate of 0.0004 can be used. After every 5 iterations, the learning rate can be adjusted to 0.1 times the original value, thus performing multiple rounds (e.g., 10 rounds) of fine-tuning training. Alternatively, other learning rates can be used during training; this application does not impose any restrictions on this.

[0223] Through the above process, it can be seen that this application can first use the overall loss of image generation of the initial denoising model itself (such as the initial image generation loss) to fine-tune the initial denoising model to obtain the denoising model. Then, the denoising model can be further strengthened by multi-step and multi-effect loss feedback training, so as to train a denoising model with excellent performance. The trained denoising model can generate an image that strongly matches the text content of the input text and is highly aesthetic.

[0224] Please see Figure 11 , Figure 11 This is a schematic diagram of an architecture for feedback training of a denoising model provided in an embodiment of this application. For example... Figure 11As shown, the training image-text pair here is the sample data pair mentioned above. This sample data pair may contain sample text and sample images that match the text content of the sample text.

[0225] The denoising model in this application can be a diffusion model. The processing device can perform feature encoding on the sample image to generate image-encoded features, and can add random sample noise maps to these image-encoded features to generate an initial noise map. Then, the processing device can perform a diffusion process on this initial noise map using the diffusion model to generate a first noise map to be denoised. During training, generating the noise map to be denoised (such as the first noise map) using sample images allows the denoising model to learn more features about the image, thereby further improving the training effect of the denoising model.

[0226] Here, the U-Net module in the diffusion model can include four QKV structures (Q1K1V1, Q2K2V2, Q3K3V3, and Q4K4V4). The example used here is training the denoising model using feedback loss of both image-text matching and image aesthetics types. The processing device can perform N-step denoising on the first noisy image using this U-Net module in the diffusion model, obtaining the denoising features Zt1 corresponding to the i-th step, Zt2 corresponding to the j-th step, and Zt0 corresponding to the N-th step.

[0227] The processing device can generate the image generation loss Loss of the denoising model using the denoising feature Zt0 and the sample image. It can also generate the feedback loss Evalloss1 (image-text matching type) for the i-th denoising step based on the denoising model, and the feedback loss Evalloss2 (image aesthetics type) for the j-th denoising step based on the denoising model, using the denoising feature Zt2. Therefore, the processing device can combine (e.g., add) the image generation loss Loss with the feedback losses Evalloss1 and Evalloss2 to obtain M total feedback losses. These M total feedback losses can then be fed back to the diffusion model to correct its model parameters (mainly including correcting the model parameters of the U-Net module), ultimately resulting in a trained denoising model (i.e., a trained diffusion model). Optionally, the image decoder in this application can be a pre-trained model that does not require further training. However, in some application scenarios, the image decoder can be trained together with the denoising model. The specific training process can be determined according to the actual application scenario. The same applies to the image encoder and text encoder.

[0228] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a model processing device provided in an embodiment of this application. Figure 12 As shown, the model processing device 120 may include: a first acquisition module 1201, a noise reduction module 1202, a second acquisition module 1203, a feedback module 1204, and a correction module 1205.

[0229] The first acquisition module 1201 is used to acquire sample data pairs, which include sample text and sample images that match the text content of the sample text.

[0230] The noise reduction module 1202 is used to add noise to the sample image to generate a first noise map, and call the noise reduction model to perform N-step noise reduction on the first noise map based on the sample text, generating noise reduction features corresponding to each step of the noise reduction process, where N is a positive integer.

[0231] The second acquisition module 1203 is used to acquire M denoising features from the N denoising features corresponding to the N-step denoising process, which are used for loss feedback of the denoising model. M is a positive integer and M is less than N.

[0232] Feedback module 1204 is used to perform loss feedback processing based on M denoising features and generate feedback loss for the denoising model. Each of the M denoising features generates a type of feedback loss.

[0233] The correction module 1205 is used to correct the model parameters of the denoising model based on the generated M feedback losses to obtain the trained denoising model. The trained denoising model is used to generate an image that matches the text content of the input text.

[0234] Optionally, the noise reduction module 1202 performs noise reduction processing on the sample image to generate a first noise map in the following ways:

[0235] The image encoder is invoked to perform feature encoding processing on the sample image, generating the image encoding features of the sample image;

[0236] The image encoding features of the sample image are noise-added to generate an initial noise map;

[0237] The noise reduction model is invoked to perform feature mapping on the initial noise map, generating the first noise map.

[0238] Optionally, the second acquisition module 1203 acquires M denoising features from the N denoising features corresponding to the N-step denoising process, for loss feedback of the denoising model, in a manner including:

[0239] Get the set first step number i and second step number j; i and j are both positive integers, and i and j are less than or equal to N respectively;

[0240] The noise reduction features corresponding to the i-th step and the j-th step are determined as M noise reduction features from the N noise reduction features;

[0241] Where i is less than j, the denoising feature corresponding to the i-th denoising step is used to generate the feedback loss of the image-text matching type, and the denoising feature corresponding to the j-th denoising step is used to generate the feedback loss of the image aesthetic type.

[0242] Optionally, the loss feedback processing includes image-text matching processing. The feedback module 1204 performs loss feedback processing based on M denoising features to generate feedback loss for the denoising model, including:

[0243] The image decoder is invoked to perform feature decoding on the denoising features corresponding to the denoising process in step i, and the first predicted image is generated.

[0244] Perform image-text matching processing on the first predicted image and the sample text to obtain the image-text matching degree between the first predicted image and the sample text.

[0245] Image-text matching loss is used to generate a noise reduction model based on image-text matching degree; image-text matching loss is a type of loss that belongs to the image-text matching property type.

[0246] Based on the image-text matching loss, the feedback loss for the denoising model in the i-th step of the denoising process is generated.

[0247] Optionally, the feedback module 1204 performs image-text matching processing on the first predicted image and the sample text to obtain the image-text matching degree between the first predicted image and the sample text, including:

[0248] The image processing model is invoked to perform feature transformation processing on the first predicted image, generating the transformed image features of the first predicted image;

[0249] The image processing model is invoked to perform feature transformation on the sample text, generating transformed text features of the sample text; the transformed image features and the transformed text features are in the same feature space of the image processing model;

[0250] Calculate the feature matching degree between the transformed image features and the transformed text features, and use the feature matching degree as the image-text matching degree;

[0251] The image-text matching process includes feature transformation processing of the first predicted image, feature transformation processing of the sample text, and calculation of the feature matching degree between the transformed image features and the transformed text features.

[0252] Optionally, the first noise map is obtained by adding a sample noise map to the image coding features of the sample image;

[0253] Feedback module 1204, based on image-text matching loss, generates the feedback loss for the denoising model in the i-th step of denoising processing, including:

[0254] Calculate the difference between the noise reduction feature corresponding to the i-th step noise reduction process and the first noise map, and use it as the predicted noise map corresponding to the i-th step noise reduction process.

[0255] Based on the difference between the predicted noise map and the sample noise map corresponding to the i-th step of noise reduction, the first noise prediction loss is generated.

[0256] The first noise prediction loss and the image-text matching loss are summed to generate the feedback loss for the denoising model in the i-th step of denoising processing.

[0257] Optionally, the loss feedback processing includes evaluation of image aesthetics. The feedback module 1204 performs loss feedback processing based on M denoising features to generate a feedback loss for the denoising model, including:

[0258] The image decoder is invoked to perform feature decoding on the denoising features corresponding to the j-th step of denoising, and the second predicted image is generated.

[0259] The aesthetic evaluation model is called to evaluate the aesthetics of the second predicted image, generate an aesthetic score for the second predicted image, and generate the aesthetic loss of the denoising model based on the aesthetic score; the aesthetic loss belongs to the loss of image aesthetics.

[0260] Based on the aesthetic loss, the feedback loss of the j-th step noise reduction process for the noise reduction model is generated.

[0261] Optionally, the first noise map is obtained by adding a sample noise map to the image coding features of the sample image;

[0262] Feedback module 1204 generates the feedback loss for the denoising model in step j based on aesthetic loss, including:

[0263] Calculate the difference between the denoised feature corresponding to the j-th denoising step and the first noise map, and use it as the predicted noise map corresponding to the j-th denoising step.

[0264] Based on the difference between the predicted noise map and the sample noise map corresponding to the j-th step of noise reduction, a second noise prediction loss is generated.

[0265] The second noise prediction loss and aesthetic loss are summed to generate the feedback loss of the noise reduction model for the j-th step of noise reduction processing.

[0266] Optionally, the correction module 1205 corrects the model parameters of the denoising model based on the generated M feedback losses to obtain the trained denoising model in the following ways:

[0267] Obtain the image generation loss of the noise reduction model;

[0268] The image generation loss is summed with each of the M feedback losses to generate the total M feedback losses.

[0269] The model parameters of the denoising model are successively corrected based on the total loss of M feedbacks to obtain the trained denoising model.

[0270] Optionally, the method by which the correction module 1205 obtains the image generation loss of the denoising model includes:

[0271] The image decoder is invoked to perform feature decoding on the denoising features corresponding to the Nth denoising step in the N-step denoising process, and a third prediction image is generated.

[0272] Based on the difference between the third predicted image and the sample image, the image generation loss of the denoising model is generated.

[0273] Optionally, the correction module 1205 sums the image generation loss with each of the M feedback losses to generate the M total feedback losses, including:

[0274] Obtain the first weight coefficient corresponding to the image generation loss, and the second weight coefficient corresponding to each feedback loss;

[0275] Based on the first weight coefficient and the second weight coefficient corresponding to each feedback loss, the image generation loss is summed with each feedback loss to generate M total feedback losses.

[0276] Optionally, the model processing device 120 further includes a fine-tuning module 1206, which is used for:

[0277] Obtain the initial noise reduction model and add noise to the sample images to generate a second noise map;

[0278] The initial denoising model is invoked to denoise the second noise map based on the sample text, generating sample denoising features;

[0279] The image decoder is invoked to perform feature decoding on the sample denoising features, generating a sample prediction image;

[0280] Based on the difference between the predicted image and the sample image, the model parameters of the initial denoising model are corrected to obtain the denoising model.

[0281] Optionally, the model processing device 120 further includes a business module 1207, which is used for:

[0282] Obtain the image generation request submitted by the business client. The image generation request carries the target text.

[0283] The target text is input into the trained denoising model, which then generates a target image that matches the text content of the target text.

[0284] The target image is returned to the business client, enabling the business client to output the target image.

[0285] Optionally, the business module 1207 may use the trained denoising model to generate a target image that matches the text content of the target text, including:

[0286] The trained noise reduction model is invoked to randomly generate a third noise map;

[0287] The trained denoising model is invoked to denoise the third noise map based on the target text, generating the target denoising features;

[0288] The image decoder is invoked to perform feature decoding processing on the target noise reduction features, generating the target image.

[0289] According to one embodiment of this application, Figure 3 The steps involved in the model processing method shown can be derived from... Figure 12 The various modules in the model processing device 120 shown are used to execute this process. For example, Figure 3 Step S101 shown can be performed by Figure 12 The first acquisition module 1201 in the process is executed. Figure 3 Step S102 shown can be performed by Figure 12 The noise reduction module 1202 in the middle is used to perform this; Figure 3 Step S103 shown can be performed by Figure 12 The second acquisition module 1203 in the middle is used to execute, Figure 3 Step S104 shown can be derived from Figure 12 The feedback module 1204 in the middle is used to execute, Figure 3 Step S105 shown can be performed by Figure 12 The correction module 1205 in the middle is used to execute it.

[0290] This application can acquire sample data pairs, including sample text and sample images matching the text content of the sample text; it can add noise to the sample images to generate a first noise map, and call a denoising model to perform N-step denoising on the first noise map based on the sample text, generating denoising features corresponding to each denoising step, where N is a positive integer; thus, from the N denoising features corresponding to the N-step denoising, M denoising features for loss feedback to the denoising model can be obtained, where M is a positive integer and M is less than N; this application can also perform loss feedback processing based on the M denoising features to generate feedback loss for the denoising model, where each of the M denoising features corresponds to a type of feedback loss; the model parameters of the denoising model are corrected based on the generated M feedback losses to obtain a trained denoising model, which is used to generate an image matching the text content of the input text. Therefore, the device proposed in this application can select denoising features (such as the selected M denoising features) corresponding to some steps of denoising processing in multiple steps of denoising processing of a noise map (such as the first noise map) (such as N steps of denoising processing) to generate different types of feedback losses for the denoising model. Thus, the model parameters of the denoising model can be more comprehensively and accurately corrected through the generated various types of feedback losses, and finally a trained denoising model can be obtained. Through the trained denoising model, an image that matches the text content of the input text and has excellent effect can be generated.

[0291] According to one embodiment of this application, Figure 12 The modules in the model processing apparatus 120 shown can be individually or entirely combined into one or more units, or some of the units can be further divided into multiple functionally smaller sub-units to achieve the same operation without affecting the technical effects of the embodiments of this application. The above modules are based on logical function division. In practical applications, the function of one module can also be implemented by multiple units, or the function of multiple modules can be implemented by one unit. In other embodiments of this application, the model processing apparatus 120 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0292] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0293] According to one embodiment of this application, a computer program capable of executing the steps involved in the corresponding methods shown in the various embodiments of this application can be run on a general-purpose computer device (which may include processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM)) to construct, as described in the embodiments of this application. Figure 12 The model processing apparatus 120 shown above. The computer program described above can be recorded on a computer-readable recording medium, and can be loaded into the computer device described above through the computer-readable recording medium and run therein.

[0294] Please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 13 As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. In some embodiments, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 13 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0295] exist Figure 13 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call computer programs stored in the memory 1005, so that the computer device 1000 described in the embodiments of this application can execute the model processing methods described in the embodiments of this application, and can also execute the methods described above. Figure 12 The description of the model processing apparatus 120 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0296] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program. When a processor executes this computer program, it can perform the model processing methods described in the various embodiments of this application; therefore, they will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0297] As an example, the aforementioned computer program can be deployed and executed on a single computer device, or deployed and executed on multiple computer devices located in one location, or executed on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can form a blockchain network.

[0298] The aforementioned computer-readable storage medium can be an internal storage unit of the computer device, such as a hard drive or memory. It can also be an external storage device, such as a plug-in hard drive, smart media card (SMC), secure digital card (SD) card, or flash card. Furthermore, the computer-readable storage medium can include both internal and external storage units of the computer device. This computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. It can also be used to temporarily store data that has been output or will be output.

[0299] This application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the model processing methods described in the embodiments of this application; therefore, these will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0300] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0301] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0302] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A model processing method, characterized in that, The method includes: Obtain sample data pairs, wherein the sample data pairs include sample text and sample images that match the text content of the sample text; The sample image is subjected to noise processing to generate a first noise map, and the noise reduction model is called to perform N-step noise reduction processing on the first noise map based on the sample text, generating noise reduction features corresponding to each step of noise reduction processing, where N is a positive integer; From the N noise reduction features corresponding to the N-step noise reduction process, obtain M noise reduction features for loss feedback of the noise reduction model, where M is a positive integer and M is less than N; Based on the M noise reduction features, loss feedback processing is performed to generate a feedback loss for the noise reduction model. Each of the M noise reduction features corresponds to a type of feedback loss. The model parameters of the denoising model are corrected based on the generated M feedback losses to obtain a trained denoising model, which is used to generate an image that matches the text content of the input text. The step of obtaining M denoising features from the N denoising features corresponding to the N-step denoising process for loss feedback of the denoising model includes: Get the set first step number i and second step number j, where i and j are both positive integers, i is less than j, and i and j are less than or equal to N respectively; The noise reduction features corresponding to the i-th and j-th noise reduction processes among the N noise reduction features are determined as the M noise reduction features. The noise reduction feature corresponding to the i-th noise reduction process is used to generate a feedback loss of image-text matching type, and the noise reduction feature corresponding to the j-th noise reduction process is used to generate a feedback loss of image aesthetic type.

2. The method as described in claim 1, characterized in that, The step of adding noise to the sample image to generate a first noise map includes: The image encoder is invoked to perform feature encoding processing on the sample image, generating the image encoding features of the sample image; The image encoding features of the sample image are subjected to noise processing to generate an initial noise map; The noise reduction model is invoked to perform feature mapping processing on the initial noise map to generate the first noise map.

3. The method as described in claim 1, characterized in that, The loss feedback processing includes image-text matching processing. The loss feedback processing based on the M denoising features to generate a feedback loss for the denoising model includes: The image decoder is invoked to perform feature decoding processing on the denoising features corresponding to the i-th step of denoising processing to generate the first predicted image; The image-text matching process is performed on the first predicted image and the sample text to obtain the image-text matching degree between the first predicted image and the sample text. The image-text matching loss of the noise reduction model is generated based on the image-text matching degree; the image-text matching loss belongs to the image-text matching type of loss. Based on the image-text matching loss, the feedback loss of the i-th step of noise reduction processing for the noise reduction model is generated.

4. The method as described in claim 3, characterized in that, The step of performing image-text matching processing on the first predicted image and the sample text to obtain the image-text matching degree between the first predicted image and the sample text includes: The image processing model is invoked to perform feature transformation processing on the first predicted image, generating transformed image features of the first predicted image; The image processing model is invoked to perform feature transformation processing on the sample text, generating transformed text features of the sample text; the transformed image features and the transformed text features are located in the same feature space of the image processing model; Calculate the feature matching degree between the converted image features and the converted text features, and use the feature matching degree as the image-text matching degree; The image-text matching process includes feature transformation processing of the first predicted image, feature transformation processing of the sample text, and calculation of the feature matching degree between the transformed image features and the transformed text features.

5. The method as described in claim 3, characterized in that, The first noise map is obtained by adding a sample noise map to the image coding features of the sample image; The step of generating the feedback loss for the denoising model in the i-th step of the denoising process, based on the image-text matching loss, includes: Calculate the difference between the noise reduction feature corresponding to the i-th step noise reduction process and the first noise map, and use it as the predicted noise map corresponding to the i-th step noise reduction process. Based on the difference between the predicted noise map corresponding to the i-th step noise reduction process and the sample noise map, a first noise prediction loss is generated; The first noise prediction loss and the image-text matching loss are summed to generate the feedback loss of the i-th step noise reduction process for the noise reduction model.

6. The method as described in claim 1, characterized in that, The loss feedback processing includes an evaluation of image aesthetics. The loss feedback processing based on the M denoising features to generate a feedback loss for the denoising model includes: The image decoder is invoked to perform feature decoding processing on the denoising features corresponding to the j-th step of denoising processing to generate the second predicted image; An aesthetic evaluation model is invoked to evaluate the aesthetic appeal of the second predicted image, generating an aesthetic appeal score for the second predicted image. Based on the aesthetic appeal score, an aesthetic appeal loss for the denoising model is generated. The aesthetic appeal loss is a loss belonging to the image aesthetic appeal type. Based on the aesthetic loss, the feedback loss of the j-th step noise reduction process for the noise reduction model is generated.

7. The method as described in claim 6, characterized in that, The first noise map is obtained by adding a sample noise map to the image coding features of the sample image; The step of generating the feedback loss for the noise reduction model in the j-th step, based on the aesthetic loss, includes: Calculate the difference between the noise reduction feature corresponding to the j-th step noise reduction process and the first noise map, and use it as the predicted noise map corresponding to the j-th step noise reduction process. Based on the difference between the predicted noise map corresponding to the j-th step noise reduction process and the sample noise map, a second noise prediction loss is generated; The second noise prediction loss and the aesthetic loss are summed to generate the feedback loss of the j-th step noise reduction process for the noise reduction model.

8. The method as described in claim 1, characterized in that, The process of correcting the model parameters of the denoising model based on the generated M feedback losses to obtain the trained denoising model includes: Obtain the image generation loss of the denoising model; The image generation loss is summed with each of the M feedback losses to generate M total feedback losses. Based on the total loss of the M feedbacks, the model parameters of the denoising model are adjusted sequentially to obtain the trained denoising model.

9. The method as described in claim 8, characterized in that, The process of obtaining the image generation loss of the denoising model includes: The image decoder is invoked to perform feature decoding processing on the denoising features corresponding to the Nth denoising step in the N-step denoising process, and a third predicted image is generated. The image generation loss of the denoising model is generated based on the difference between the third predicted image and the sample image.

10. The method as described in claim 8, characterized in that, The process of summing the image generation loss with each of the M feedback losses to generate M total feedback losses includes: Obtain the first weight coefficient corresponding to the image generation loss, and the second weight coefficient corresponding to each feedback loss; Based on the first weight coefficient and the second weight coefficient corresponding to each feedback loss, the image generation loss is summed with each feedback loss to generate the M total feedback losses.

11. The method as described in claim 1, characterized in that, The process of obtaining the noise reduction model includes: An initial noise reduction model is obtained, and the sample image is subjected to noise addition processing to generate a second noise map; The initial denoising model is invoked to denoise the second noise map based on the sample text, generating sample denoising features; The image decoder is invoked to perform feature decoding processing on the denoising features of the sample, generating a sample prediction image; Based on the difference between the predicted image and the sample image, the model parameters of the initial denoising model are corrected to obtain the denoising model.

12. The method as described in claim 1, characterized in that, The method further includes: Obtain an image generation request submitted by the business client, wherein the image generation request carries target text; The target text is input into the trained denoising model to generate a target image that matches the text content of the target text. The target image is returned to the service client, so that the service client outputs the target image.

13. The method as described in claim 12, characterized in that, The step of calling the trained noise reduction model to generate a target image that matches the text content of the target text includes: The trained noise reduction model is invoked to randomly generate a third noise map; The trained denoising model is invoked to denoise the third noise map based on the target text, generating target denoising features; The image decoder is invoked to perform feature decoding processing on the target noise reduction features to generate the target image.

14. A model processing device, characterized in that, The device includes: The first acquisition module is used to acquire sample data pairs, wherein the sample data pairs include sample text and sample images that match the text content of the sample text; The noise reduction module is used to add noise to the sample image to generate a first noise map, and call the noise reduction model to perform N-step noise reduction processing on the first noise map based on the sample text, generating noise reduction features corresponding to each step of noise reduction processing, where N is a positive integer; The second acquisition module is used to acquire M noise reduction features from the N noise reduction features corresponding to the N-step noise reduction process, which are used for loss feedback of the noise reduction model, where M is a positive integer and M is less than N. The feedback module is used to perform loss feedback processing based on the M noise reduction features to generate a feedback loss for the noise reduction model. Each of the M noise reduction features corresponds to a type of feedback loss. The correction module is used to correct the model parameters of the denoising model based on the generated M feedback losses to obtain a trained denoising model, which is used to generate an image that matches the text content of the input text. The process by which the second acquisition module acquires the M noise reduction features includes: Get the set first step number i and second step number j, where i and j are both positive integers, i is less than j, and i and j are less than or equal to N respectively; The noise reduction features corresponding to the i-th and j-th noise reduction processes among the N noise reduction features are determined as the M noise reduction features. The noise reduction feature corresponding to the i-th noise reduction process is used to generate a feedback loss of image-text matching type, and the noise reduction feature corresponding to the j-th noise reduction process is used to generate a feedback loss of image aesthetic type.

15. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-13.

16. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1-13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and to execute the steps of the method according to any one of claims 1-13.

Images