Image generation model training method and device, image processing method and device

Abstract

The present disclosure provides an image generation model training method and device, which relates to the field of artificial intelligence technology, specifically to the technical fields of computer vision, deep learning, large models, etc., and can be applied to scenarios such as artificial intelligence content generation. The specific implementation scheme is: obtain an image sample set; obtain a pre-built image generation network, the image generation network includes: a sequentially connected image and text recognition module, a large language model, and a text generation image model; input the image sample selected from the image sample set into the image generation network to obtain the generated image output by the image generation network; use the image scoring model to score the generated image to obtain the evaluation value of the generated image; based on the evaluation value, calculate the network loss value of the image generation network; based on the network loss value of the image generation network, train the image generation network to obtain a trained image generation model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as artificial intelligence content generation, and in particular to an image generation model training method and device, an image processing method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the introduction of the SD (stable diffusion) model, generative image models have demonstrated powerful capabilities, including more realistic scenes, richer details, and good command-following capabilities.

[0003] For the widely used deep learning text-to-image generation model, inputting the prompt word into the deep learning text-to-image generation model can generate almost any image that humans can imagine. Summary of the invention

[0004] The present disclosure provides an image generation model training method and device, an image processing method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to a first aspect, a method for training an image generation model is provided, the method comprising: obtaining an image sample set, the image sample set comprising at least one image sample; obtaining a pre-constructed image generation network, the image generation network comprising: a picture-text recognition module, a large language model and a text generation image model connected in sequence, the picture-text recognition module obtaining a recognition text based on an input image; the large language model obtaining a prompt word text with multi-image description features based on the recognition text; the text generation image model obtaining a generated image based on the prompt word text; performing the following general training steps: inputting an image sample selected from the image sample set into the image generation network to obtain a generated image output by the image generation network; using an image scoring model to score the generated image to obtain an evaluation value of the generated image; calculating a network loss value of the image generation network based on the evaluation value; training the image generation network based on the network loss value of the image generation network to obtain a trained image generation model.

[0006] According to a second aspect, an image processing method is provided, the method comprising: acquiring an image to be processed; inputting the image to be processed into an image generation model generated by the method described in any implementation manner of the first aspect, to obtain an image generation result of the image to be processed.

[0007] According to a third aspect, an image generation model training device is provided, the device comprising: a set acquisition unit, configured to acquire an image sample set, the image sample set comprising at least one image sample; a network acquisition unit, configured to acquire a pre-constructed image generation network, the image generation network comprising: a picture and text recognition module, a large language model and a text generation image model connected in sequence, the picture and text recognition module obtains a recognition text based on an input image; the large language model obtains a prompt word text with multi-image description features based on the recognition text; the text generation image model obtains a generated image based on the prompt word text; a sample input unit, configured to input an image sample selected from the image sample set into the image generation network, to obtain a generated image output by the image generation network; a scoring unit, configured to score the generated image using the image scoring model, to obtain an evaluation value of the generated image; a calculation unit, configured to calculate a network loss value of the image generation network based on the evaluation value; a model acquisition unit, configured to train the image generation network based on the network loss value of the image generation network, to obtain a trained image generation model.

[0008] According to the fourth aspect, an image processing device is provided, which includes: an image acquisition unit, configured to acquire an image to be processed; a result obtaining unit, configured to input the image to be processed into an image generation model generated by the device described in any implementation method of the third aspect, and output an image generation result of the image to be processed.

[0009] According to the fifth aspect, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation of the first aspect or the second aspect.

[0010] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect or the second aspect.

[0011] According to a seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method described in any implementation of the first aspect or the second aspect.

[0012] The image generation model training method and device provided by the embodiments of the present disclosure are as follows: first, an image sample set is obtained, the image sample set includes at least one image sample; second, a pre-constructed image generation network is obtained, the image generation network includes: a picture and text recognition module, a large language model and a text generation image model connected in sequence, the picture and text recognition module obtains a recognition text based on an input image; the large language model obtains a prompt word text with multi-image description features based on the recognition text; the text generation image model obtains a generated image based on the prompt word text; third, the image sample selected from the image sample set is input into the image generation network to obtain a generated image output by the image generation network; third, the image scoring model is used to score the generated image to obtain an evaluation value of the generated image; then, based on the evaluation value, the network loss value of the image generation network is calculated; finally, based on the network loss value of the image generation network, the image generation network is trained to obtain a trained image generation model. The present invention generates prompt word text for multiple image description features of recognized text through a large language model, thereby improving the guidance of prompt words to the text generation image model; in the image generation network training process, the generated image is scored through an image scoring model, and the image generation model is obtained based on the evaluation value obtained by the scoring, thereby improving the reliability and accuracy of the image generation model training and improving the effect of the model in image generation.

[0013] The image processing method and device provided by the embodiments of the present disclosure obtain an image to be processed, input the image to be processed into an image generation model generated by an image generation model training method, and obtain an image generation result of the image to be processed. Thus, the image generation result is generated by using an image generation model including a large language model, which improves the reliability and accuracy of the image generation result.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0016] Figure 1 is a flowchart of an embodiment of the image generation model training method according to the present disclosure;

[0017] Figure 2 is a structural schematic diagram of image generation network training in an embodiment of the present disclosure;

[0018] Figure 3 is a flow chart of an embodiment of an image processing method according to the present disclosure;

[0019] Figure 4 is a flowchart of another embodiment of the image processing method in the embodiment of the present disclosure;

[0020] Figure 5 is a structural schematic diagram of an embodiment of an image generation model training device according to the present disclosure;

[0021] Figure 6 is a schematic structural diagram of an embodiment of an image processing device according to the present disclosure;

[0022] Figure 7 It is a block diagram of an electronic device used to implement the image generation model training method or image processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] The traditional style image generation process includes: selecting a reference image and using the original image to directly guide the generation of a new image. The original image is transformed to obtain the edge distribution map of the image content, the approximate color value distribution, and the overall composition information, which are input into the image generation model together with the text. This information comprehensively guides the generated image. This image should have the style or content characteristics that you want to see in the generated image.

[0025] Existing solutions require the original image to be preprocessed and then input into the original generation model. However, these preprocessings may not fully reflect the content of the image, especially abstract features such as image style. Humanistic content cannot be well guided in this way. In content-guided scenarios, the model may not be able to accurately reproduce the details in the reference image, resulting in the generated image deviating from the original image in content. Or it follows the original image too closely, without more divergence and extension effects. The generation results are highly dependent on the text prompts provided by the user and the quality of the reference image. Inaccurate or ambiguous input may lead to unsatisfactory output.

[0026] Based on this, the present invention proposes an image generation model training method. Figure 1 A process 100 according to an embodiment of the image generation model training method of the present disclosure is shown. The image generation model training method comprises the following steps:

[0027] Step 101: Obtain an image sample set.

[0028] In this embodiment, the execution subject on which the image generation model training method runs can obtain the image sample set in a variety of ways. For example, the execution subject can obtain the image sample set stored in the database server through a wired connection or a wireless connection. For another example, the user can obtain the image sample set collected by the terminal by communicating with the terminal.

[0029] Here, the image sample set may include at least one image sample, the image sample includes an annotated image related to the generated image to be generated, and the content displayed in the annotated image includes various objects, scenes and styles; optionally, the image sample may also include image description text. When training the image generation network, the image in the image sample can be input into the image-text recognition module, the image description text can be spliced ​​with the recognition text, and the spliced ​​text can be input into the large language model.

[0030] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of video frames and video sequences involved are carried out after authorization and comply with relevant laws and regulations.

[0031] Step 102, obtaining a pre-built image generation network.

[0032] Among them, the image generation network includes: a picture and text recognition module, a large language model and a text generation image model connected in sequence. The picture and text recognition module obtains the recognized text based on the input image; the large language model obtains the prompt word text with multi-image description features based on the recognized text; the text generation image model obtains the generated image based on the prompt word text.

[0033] In this embodiment, the image-text recognition module is an image content extractor that converts an image into text. The image-text recognition module can be a multimodal image-text recognition model. The image is input into the multimodal image-text recognition model to obtain the recognition text that describes the content in the image output by the multimodal image-text recognition model. The recognition text is information that represents the image in the form of text data. The multimodal image-text recognition model can be trained to support multiple languages, thereby overcoming language barriers.

[0034] In this embodiment, the big language model is a natural language processing model based on deep learning. It can automatically generate sentences, paragraphs or articles that conform to language rules mainly by learning a large amount of text data. The core idea of ​​the big language model is to use deep neural networks to learn the grammar, semantics and other features of natural language, so as to predict the probability of the next word to appear and generate new sentences based on these probabilities.

[0035] In this embodiment, the recognition text is input into the large language model to obtain a prompt word text of multiple image description features output by the large language model, wherein the image description feature is a text describing the image feature. The prompt word text obtained by the large language model disclosed in the present invention has multiple image description features. Through multiple image description features, the input image of the input image and text recognition module can be described more comprehensively.

[0036] In this embodiment, the text-generated image model is a model based on text-generated style images. By inputting the prompt word text into the text-generated image model, the image generation requirements can be input into the text-generated image module, thereby providing the text-generated image module with more target image requirements.

[0037] In this embodiment, the text-to-image model can be an SD (stable diffusion) model. The recognition text generated by the image-text recognition module is relatively detailed and accurate. At the same time, the prompt word text automatically generated by the large language model may provide new perspectives and creativity, providing more possible space for the image generation of the text-to-image model, and even inspiring the creative inspiration of the model users.

[0038] In step 103, the image samples selected from the image sample set are input into the image generation network to obtain the generated image output by the image generation network.

[0039] In this embodiment, the execution subject can select image samples from the image sample set obtained in step 101, and execute the training steps from step 103 to step 106 to complete an iterative training of the image generation network. Among them, the selection method and the number of video frames selected from the image sample set are not limited in this application, and the number of iterative training of the image generation network is not limited. For example, in one iterative training, multiple continuous image sets can be randomly selected, and the selected image sets can have only images or corresponding description texts of the images. The network loss value of the image generation network is calculated through the selected image samples, and the parameters of the image generation network are adjusted.

[0040] Step 104: Use the image scoring model to score the generated image to obtain an evaluation value of the generated image.

[0041] In this embodiment, the image scoring model is a pre-trained model for scoring the content and form of expression of an image. By inputting the image into the image scoring model, an evaluation value output by the image scoring model can be obtained, and the evaluation value is a specific representation of the richness and beauty of the image content.

[0042] In this embodiment, the image scoring model can be an image-to-text conversion model obtained by training a large model, which refers to a deep learning or machine learning model with a large number of parameters, which can be automatically adjusted through a training process to capture complex relationships in the input data. Such models usually have a deeper network structure and more neurons to increase the representation and learning capabilities of the model.

[0043] Specific as Figure 2 As shown, the sample image is input into the image-text recognition module, the image-text recognition module outputs the recognized text, the large language model obtains the prompt word text based on the recognized text, the text-to-image generation model obtains the generated image based on the prompt word text, the image scoring model obtains the generated image, scores the generated image to obtain the evaluation value, and the image generation network is trained through the evaluation value.

[0044] Step 105: Calculate the network loss value of the image generation network based on the evaluation value.

[0045] In this embodiment, during each iterative training of the image generation network, image samples are selected from the image sample set, and the selected image samples are input into the image generation network. Based on the loss function and evaluation value pre-set for the image generation network, the network loss value of the image generation network is calculated.

[0046] In this embodiment, the loss function of the image generation network can adopt a mean square error function, which is the expectation of the square of the difference between the predicted value (estimated value) of the image generation network and the true value. During the iterative training process of the image generation network, the gradient descent algorithm can be used to minimize the loss function of the image generation network, thereby iteratively optimizing the network parameters of the image generation network.

[0047] The original meaning of gradient is a vector, which means that the directional derivative of a loss function at that point reaches the maximum value along that direction, that is, the loss function changes fastest along that direction at that point and the rate of change is the largest. In deep learning, the main task of neural networks is to find the optimal network parameters (weights and biases) during learning. The optimal network parameters are also the parameters when the loss function is minimized.

[0048] During the image generation network training process, a loss function can be designed only for the text generation image model. The loss value can be calculated through the loss function of the text generation image model. Based on the loss value and the evaluation value, the gradient of the large language model is updated and the parameters of the large language model are adjusted. There is no need to adjust the parameters of the text generation image model, thereby achieving the goal of hot-swapping the text generation image model.

[0049] Optionally, during the image generation network training process, a loss function can be designed only for the text generation image model. The loss value is calculated through the loss function of the text generation image model. Based on the loss value and the evaluation value, the parameters of the large language model and the text generation image model are updated to achieve the purpose of adjusting both the text generation image model and the large language model at the same time.

[0050] In this embodiment, the above-mentioned calculation of the network loss value of the image generation network based on the evaluation value includes: calculating the overall loss value of the image generation network; dividing the overall loss value by the evaluation value to obtain the network loss value.

[0051] Step 106: train the image generation network based on the network loss value of the image generation network to obtain a trained image generation model.

[0052] In this embodiment, the image generation model is a trained image generation network obtained after adjusting the parameters of the image generation network through multiple iterative training. The network loss value of the image generation network can be used to detect whether the image generation network meets the training completion conditions. After the image generation network meets the training completion conditions, the image generation model is obtained.

[0053] Optionally, in this embodiment, in response to the image generation network not satisfying the training completion condition, the relevant parameters in the image generation network are adjusted so that the network loss value of the image generation network converges, and the above-mentioned training steps 103-106 are continued to be executed based on the adjusted image generation network.

[0054] In this optional implementation, when the image generation network does not meet the training completion conditions, adjusting the relevant parameters of the image generation network helps to help the network loss value of the image generation network converge.

[0055] The image generation model training method provided in this embodiment can automatically generate prompt word text, greatly reducing the time and labor of manual input, especially for large-scale image processing tasks, improving the efficiency of image processing tasks. In addition, the automatically generated prompt word text can provide standardized image descriptions, which is helpful for unified communication and search of image content.

[0056] The image generation model training method provided by the embodiment of the present disclosure comprises the following steps: first, obtaining an image sample set, the image sample set comprising at least one image sample; second, obtaining a pre-built image generation network, the image generation network comprising: a picture-text recognition module, a large language model and a text generation image model connected in sequence, the picture-text recognition module obtaining a recognition text based on an input image; the large language model obtaining a prompt word text having multi-image description features based on the recognition text; the text generation image model obtaining a generated image based on the prompt word text; third, inputting an image sample selected from the image sample set into the image generation network to obtain a generated image output by the image generation network; third, using an image scoring model to score the generated image to obtain an evaluation value of the generated image; then, calculating a network loss value of the image generation network based on the evaluation value; finally, training the image generation network based on the network loss value of the image generation network to obtain a trained image generation model. The present invention generates prompt word text for multiple image description features of recognized text through a large language model, thereby improving the guidance of prompt words to the text generation image model; in the image generation network training process, the generated image is scored through an image scoring model, and the image generation model is obtained based on the evaluation value obtained by the scoring, thereby improving the reliability and accuracy of the image generation model training.

[0057] In some optional implementations of the present disclosure, the above-mentioned calculation of the network loss value of the image generation network based on the evaluation value includes: obtaining the loss function of the text generation image model; calculating the model loss value of the text generation image model based on the selected image samples and the loss function; and adjusting the model loss value based on the evaluation value to obtain the network loss value.

[0058] In this optional implementation, the text-generated image model is used as the main adjustment network to obtain the loss function of the text-generated image model. The above-mentioned calculation of the loss value of the text-generated image model based on the selected image sample and the loss function includes: obtaining the generated image of the text-generated image model based on the selected image sample; obtaining the difference value between the selected image sample and the generated image based on the loss function, and using the difference value as the model loss value.

[0059] The above-mentioned adjusting the model loss value based on the evaluation value to obtain the network loss value includes: dividing the model loss value by the evaluation value to obtain the network loss value.

[0060] The method for calculating the network loss value of an image generation network provided in this embodiment obtains the loss function of a text generation image model, calculates the model loss value of the text generation image model based on selected image samples and the loss function; adjusts the model loss value based on the evaluation value to obtain the network loss value, obtains the model loss value based on the text generation image model, and adjusts the model loss value based on the evaluation value of the image scoring model to obtain the network loss value, thereby providing a reliable implementation method for obtaining the network loss value.

[0061] In some optional implementations of the present disclosure, the above-mentioned network loss value based on the image generation network is used to train the image generation network to obtain a trained image generation model, including: in response to the network loss value of the image generation network satisfying the training completion condition, using the image generation network as the image generation model.

[0062] In this optional implementation, the training completion condition includes at least one of the following: the number of training iterations of the image generation network reaches a predetermined iteration threshold, and the network loss value of the image generation network is less than a predetermined network loss value threshold. The predetermined iteration threshold is an empirical value obtained based on the network loss value of the image generation network. For example, the predetermined iteration threshold of the image generation network is 50,000 times. The predetermined network loss value threshold of the image generation network is 0.01.

[0063] The method for obtaining the image generation model provided by this optional implementation uses the image generation network as the image generation model when the network loss value of the image generation network meets the training completion condition, thereby providing a reliable implementation method for generating the image generation model.

[0064] In some optional implementations of the present disclosure, the above-mentioned image generation model training method also includes: in response to the network loss value of the image generation network not satisfying the training completion condition, adjusting the parameters of the large language model based on the network loss value, and continuing to train the image generation network.

[0065] In this embodiment, the above-mentioned continuing to execute the total training step refers to: continuing to select image samples from the image sample set, inputting the image samples selected from the image sample set into the image generation network, and obtaining a generated image output by the image generation network; using the image scoring model to score the generated image to obtain an evaluation value of the generated image; based on the evaluation value, calculating the network loss value of the image generation network; based on the network loss value of the image generation network, obtaining the image generation model.

[0066] The image generation model training method provided in this embodiment adjusts the parameters of the large language model without changing the parameters of the text generation image model when the network loss value of the image generation network does not meet the training completion conditions. The pluggable text generation image model can be applied to the image generation network, thereby improving the flexibility of the application of the text generation image model.

[0067] The image generation model training method provided in this embodiment has a small number of parameters of the large language model. When the network loss value of the image generation network does not meet the training completion conditions, only the parameters of the large language model are adjusted, and the parameters of the text generation image model are not changed. This can reduce the amount of network training and improve the convergence effect of the image generation network.

[0068] In some optional implementations of the present disclosure, the above-mentioned multi-image description features include: element features, composition features and style features.

[0069] In this optional implementation, the above-mentioned element features are feature texts describing each unit in the generated image to be generated. The unit can be a person, an object, an animal, or a scenery. For example, the recognition text includes: a rooster, and the element feature is: a rooster made of vintage tin and wood.

[0070] In this optional implementation, the composition feature is a feature text describing the layout of each unit in the generated image to be generated. For example, if the recognition text includes: a rooster in the middle of the image, the composition feature is a rooster in the center of the background under a white background.

[0071] In this optional implementation, the above-mentioned style features are features that describe the style characteristics of the generated image to be generated. For example, the style features include: retro style, mechanical style, comic style, oil painting style, etc.

[0072] Optionally, the above-mentioned multi-image description features may also include: color tone features, plot features, etc., wherein the color tone features are features that describe the color tone of the generated image to be generated, and the plot features are features that describe the storyline of the generated image to be generated.

[0073] The multi-image description features provided by this optional implementation method limit the prompt word text through element features, composition features and style features, thereby improving the richness of prompt word text generation and ensuring the reliability of the image generation model.

[0074] In some optional implementations of the present disclosure, an image scoring model is obtained by training a multimodal image and text recognition network, and the training steps of the multimodal image and text recognition network include: a first training step that can be iterated multiple times to obtain a first scoring module; then a second training step that can be iterated multiple times, the second training step is a training step performed after the first training step; by iterating the first training step multiple times, a first scoring model can be obtained; by iterating the second training step multiple times, an image scoring model can be obtained.

[0075] Specifically, the above-mentioned first training step includes: inputting the acquired first image sample and description text into the multimodal image-text recognition network to obtain the answer text output by the multimodal image-text network; splicing the first image sample, description text, answer text and scoring text to obtain first splicing information, and inputting the first splicing information into the multimodal image-text recognition network to obtain the first score output by the multimodal image-text network; based on the first score, calculating the loss value of the multimodal image-text network; based on the loss value of the multimodal image-text network, obtaining the first scoring model.

[0076] In this embodiment, the first image sample includes a first image and a score of the first image, wherein the first image may be an image generated by an image generation model, and the score of the first image may be a score obtained after manually scoring the first image. The loss value of the multimodal image-text network may be calculated through the score of the first image, the first score, and the loss function of the multimodal image-text network. In response to the loss value of the multimodal image-text network satisfying the training completion condition, a first scoring model is obtained.

[0077] In this embodiment, the description text is a text that requires the model to describe the first image in the first image sample. For example, the description text is: Please describe the main content of the first image in the form of text. The rating text is a text that requires the model to give a rating to the first image in the first image sample. For example, the rating text is: Please give a rating for the first image.

[0078] In this embodiment, the first image sample, description text, answer text and scoring text are spliced ​​to obtain first splicing information, and the first splicing information is input into a multimodal image-text recognition network. The multimodal image-text recognition network can integrate the first image, description text, answer text and scoring text in the first image sample to give a first score to the first image in the first image sample.

[0079] In this embodiment, the answer text is a text in which the multimodal image-text recognition network describes the image content of the first image in the first image sample after obtaining the description text. The comprehensiveness of the description of the first image by the multimodal image-text recognition network can be determined through the answer text.

[0080] In this embodiment, the first score is the score obtained after the multimodal image-text recognition network scores the first image in the first image sample, wherein the first score may be a score of the aesthetic effect or cognitive effect of the first image.

[0081] Specifically, the second training step includes: inputting the acquired second image sample and the scoring text into the first scoring model to obtain a second score output by the first scoring model; based on the second score, calculating the loss value of the first scoring model; based on the loss value of the first scoring model, obtaining an image scoring model.

[0082] In this embodiment, the second image sample includes a second image and a score of the second image, wherein the second image may be an image generated by an image generation model, and the score of the second image may be a score obtained after manually scoring the second image. The loss value of the first scoring model may be calculated through the score of the second image, the second score, and the loss function of the first scoring model (which is also the loss function of the multimodal image-text network). In response to the loss value of the first scoring model satisfying the training completion condition, an image scoring model is obtained.

[0083] In this embodiment, by training the second training step for multiple iterations, an image scoring model can be trained to obtain an input image and output a score for the image, thereby improving the convenience of using the image scoring model.

[0084] The method for training a multimodal image and text recognition network provided in this embodiment can, through the first training step, enable the multimodal image recognition network to fully understand the image and the image content and give a score to the image; through the second training step, a model with an image as input and a score as output can be obtained through training, thereby improving the reliability of the image scoring model.

[0085] Optionally, the image scoring model is obtained by training a multimodal image and text recognition network, and the training steps of the multimodal image and text recognition network include: inputting the acquired scoring image samples into the multimodal image and text recognition network to obtain the scores output by the multimodal image and text recognition network; in response to the multimodal image and text recognition network satisfying the training completion conditions, obtaining the image scoring model.

[0086] Furthermore, based on the image generation model training method provided in the above embodiment, the present disclosure also provides an embodiment of an image processing method. The image processing method of the present disclosure combines artificial intelligence fields such as computer vision and deep learning.

[0087] See also Figure 3 , shows a process 300 according to an embodiment of the image processing method of the present disclosure. The image processing method provided in this embodiment includes the following steps:

[0088] Step 301: Obtain an image to be processed.

[0089] In this embodiment, the image to be processed may include information such as people, objects, and scenery. The image generation model is used to process the image to be processed to obtain an image generation result. The execution subject of the image processing method can obtain the image to be processed in a variety of ways. For example, the execution subject can obtain the image to be processed stored in the database server through a wired connection or a wireless connection. For another example, the execution subject can also receive the image to be processed collected by a terminal or other device in real time.

[0090] Step 302: input the image to be processed into the image generation model, and output the image generation result of the image to be processed.

[0091] In this embodiment, the execution subject may input the image to be processed obtained in step 301 into the image generation model, thereby obtaining an image generation result of the image to be processed. It should be noted that the image generation result includes a generated image, which is a newly generated image with a style and / or content transformation relative to the image to be processed.

[0092] In this embodiment, the image generation model can be adopted as described above Figure 1 The specific training process can be found in Figure 1 The relevant description of the embodiments will not be repeated here.

[0093] The image processing method provided by the embodiment of the present disclosure obtains multiple images to be processed; the images to be processed are input into the image generation model generated by the image generation model training method of the present embodiment to obtain the image generation result of the images to be processed. Thus, the image generation model can be used to reliably process the images to be processed, thereby improving the effectiveness of image processing.

[0094] In some embodiments of the present disclosure, the above-mentioned image processing method includes: acquiring an image to be processed; detecting whether the size of the image to be processed is a standard size; in response to the size of the image to be processed not being a standard size, adjusting the image to be processed to the standard size.

[0095] In this embodiment, the standard size may be a size that matches the image generation model, for example, the standard size is 448*448.

[0096] In this embodiment, the size of the image to be processed can be directly measured by the length measurement tool in the image processing tool. When the size of the image to be processed is not a standard size, the image to be processed can be processed by the image cropping tool or the image scaling tool in the image processing tool to obtain an image to be processed with a standard size.

[0097] The image processing method provided in this embodiment adjusts the image to be processed to the standard size when the size of the image to be processed is not a standard size, which can reduce the steps of image processing by the image generation model and improve the reliability of the image generation result.

[0098] In some optional implementations of the present disclosure, the above-mentioned image generation model includes: a graphic and text recognition module, a large language model and a text-generated image model. The image to be processed is input into the image generation model, and the image generation result of the image to be processed is output, including: inputting the image to be processed into the graphic and text recognition module to obtain the recognized text output by the graphic and text recognition module; inputting the recognized text into the large language model to obtain the prompt word text output by the large language model; inputting the prompt word text into the text-generated image model to obtain the generated image output by the text-generated image model.

[0099] In this embodiment, the image and text recognition module can be a multimodal image and text recognition model, and the multimodal image and text recognition model is a model obtained by training a multimodal image and text recognition network. Specifically, the training process of the multimodal image and text recognition model is as follows: obtaining image samples from an image sample set, inputting the image samples into a multimodal image and text recognition network, and obtaining text output by the multimodal image and text recognition network; calculating the loss value of the multimodal image and text recognition network; and in response to the multimodal image recognition network satisfying its training completion conditions, obtaining the multimodal image and text recognition model.

[0100] like Figure 4 As shown, the image D to be processed is input into the image-text recognition module M1 to obtain the recognition text S output by the image-text recognition module M1, and the content of the recognition text S is "a rooster standing on the ground, with a red comb and a huge tail with plump feathers"; the recognition text S is input into the large language model M2 to obtain the prompt word text T output by the large language model M2, and the content of the prompt word text T is "a rooster made of vintage tin and wood, with a white background, a proud expression, a majestic standing posture, and a mechanical style"; the prompt word text T is input into the text generation image model M3 to obtain the generated image W output by the text generation image model M3.

[0101] The image processing method provided in this embodiment, when the image generation model includes a picture-text recognition module, a large language model and a text-to-image generation model, obtains the recognition text through the picture-text recognition module, obtains the prompt word text through the large language model, and obtains the generated image through the text-to-image generation model in sequence. Since the prompt word text of the multi-image description feature output by the large language model improves the accuracy of the generated image.

[0102] In some embodiments of the present disclosure, the above-mentioned image processing method also includes: receiving image processing requirement text; after obtaining the recognition text, splicing the recognition text with the image processing requirement text to obtain second splicing information; inputting the second splicing information into the large language model to obtain a new prompt word text output by the large language model; inputting the new prompt word text into the text generation image model to obtain a new generated image output by the text generation image model.

[0103] In this embodiment, the image processing requirement text may be a requirement for the generated image to be generated input by the user, and the specific requirement of the user may be extracted through the image processing requirement text.

[0104] The image processing method provided in this embodiment obtains an image to be processed; receives an image processing requirement text; inputs the image to be processed into a graphic and text recognition module to obtain a recognition text output by the graphic and text recognition module; splices the image processing requirement text and the recognition text to obtain second splicing information; inputs the second splicing information into a large language model to obtain a new prompt word text output by the large language model; inputs the new prompt word text into a text generation image model to obtain a new generated image output by the text generation image model.

[0105] Optionally, the above-mentioned image processing method may further include: outputting new prompt word text, so that in the user interaction environment, the model can provide immediate text feedback to enhance the user experience. The model can customize the generated text prompts according to the user's preferences and historical feedback, and can also customize more diverse or unified texts according to the image processing requirement text and business scenarios input by the user again.

[0106] The image processing method provided in this embodiment, after receiving the image processing requirement text, splices the image processing requirement text with the recognition text to obtain second splicing information; inputs the second splicing information into the large language model to obtain a new prompt word text output by the large language model; inputs the new prompt word text into the text-to-image model to obtain a new generated image output by the text-to-image model, so that the user's image processing can be obtained through the image processing requirement text, and the image generation model obtains a new generated image based on the integration of the user's processing requirements, thereby improving the accuracy of the generated image generation.

[0107] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an image generation model training device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0108] like Figure 5As shown, the image generation model training device 500 provided in this embodiment includes: a set acquisition unit 501, a network acquisition unit 502, a sample input unit 503, a scoring unit 504, a calculation unit 505, and a model acquisition unit 506. Among them, the above-mentioned set acquisition unit 501 can be configured to acquire an image sample set, and the image sample set includes at least one image sample. The above-mentioned network acquisition unit 502 can be configured to acquire a pre-built image generation network, and the image generation network includes: a picture and text recognition module, a large language model, and a text generation image model connected in sequence. The picture and text recognition module obtains the recognition text based on the input image; the large language model obtains the prompt word text with multi-image description features based on the recognition text; the text generation image model obtains the generated image based on the prompt word text. The above-mentioned sample input unit 503 can be configured to input the image sample selected from the image sample set into the image generation network to obtain the generated image output by the image generation network. The above-mentioned scoring unit 504 can be configured to score the generated image using the image scoring model to obtain the evaluation value of the generated image. The calculation unit 505 may be configured to calculate a network loss value of the image generation network based on the evaluation value. The model acquisition unit 506 may be configured to train the image generation network based on the network loss value of the image generation network to obtain a trained image generation model.

[0109] In this embodiment, in the image generation model training device 500, the specific processing of the set acquisition unit 501, the network acquisition unit 502, the sample input unit 503, the scoring unit 504, the calculation unit 505, and the model acquisition unit 506 and the technical effects thereof can be referred to respectively. Figure 1 The relevant descriptions of step 101, step 102, step 103, step 104, step 105, and step 106 in the corresponding embodiments are not repeated here.

[0110] In some optional implementations of this embodiment, the above-mentioned calculation unit 505 is further configured to: obtain the loss function of the text-generated image model; calculate the model loss value of the text-generated image model based on the selected image samples and the loss function; adjust the model loss value based on the evaluation value to obtain the network loss value.

[0111] In some optional implementations of this embodiment, the above-mentioned model obtaining unit 506 is further configured to: in response to the network loss value of the image generation network satisfying the training completion condition, use the image generation network as the image generation model.

[0112] In some optional implementations of this embodiment, the above-mentioned device 500 also includes: an adjustment unit (not shown in the figure), which is configured to adjust the parameters of the large language model based on the network loss value in response to the network loss value of the image generation network not satisfying the training completion condition, and control the sample input unit 503 to operate.

[0113] In some optional implementations of this embodiment, the above-mentioned multi-image description features include: element features, composition features and style features.

[0114] In some optional implementations of the present embodiment, the above-mentioned image scoring model is obtained by training a multimodal image-text recognition network, and the multimodal image-text recognition network is trained by a training unit (not shown in the figure), and the training unit is configured to: input the acquired first image sample and description text into the multimodal image-text recognition network to obtain the answer text output by the multimodal image-text network; splice the first image sample, description text, answer text and scoring text to obtain first splicing information, and input the first splicing information into the multimodal image-text recognition network to obtain the first score output by the multimodal image-text network; based on the first score, calculate the loss value of the multimodal image-text network; based on the loss value of the multimodal image-text network, obtain the first scoring model; input the acquired second image sample and scoring text into the first scoring model to obtain the second score output by the first scoring model; based on the second score, calculate the loss value of the first scoring model; based on the loss value of the first scoring model, obtain the image scoring model.

[0115] The image generation model training device provided by the embodiment of the present disclosure is as follows: first, the set acquisition unit 501 acquires an image sample set, and the image sample set includes at least one image sample; secondly, the network acquisition unit 502 acquires a pre-constructed image generation network, and the image generation network includes: a picture and text recognition module, a large language model, and a text generation image model connected in sequence, the picture and text recognition module obtains a recognition text based on the input image; the large language model obtains a prompt word text with multi-image description features based on the recognition text; the text generation image model obtains a generated image based on the prompt word text; thirdly, the sample input unit 503 inputs the image sample selected from the image sample set into the image generation network to obtain a generated image output by the image generation network; thirdly, the scoring unit 504 uses the image scoring model to score the generated image to obtain an evaluation value of the generated image; thirdly, the calculation unit 505 calculates a network loss value of the image generation network based on the evaluation value; finally, the model acquisition unit 506 trains the image generation network based on the network loss value of the image generation network to obtain a trained image generation model. The present invention generates prompt word text for multiple image description features of recognized text through a large language model, thereby improving the guidance of prompt words to the text generation image model; in the image generation network training process, the generated image is scored through an image scoring model, and the image generation model is obtained based on the evaluation value obtained by the scoring, thereby improving the reliability and accuracy of the image generation model training.

[0116] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an image processing device, which is similar to Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0117] like Figure 6 As shown, the image processing device 600 provided in this embodiment includes: an image acquisition unit 601, and a result acquisition unit 602. The image acquisition unit 601 can be configured to acquire an image to be processed. The result acquisition unit 602 can be configured to input the image to be processed into the above-mentioned Figure 5 In the image generation model generated by the device described in the embodiment, an image generation result of the image to be processed is output.

[0118] In this embodiment, in the image processing device 600, the specific processing of the image acquisition unit 601 and the result obtaining unit 602 and the technical effects thereof can be referred to in Figure 3 The relevant descriptions of step 301 and step 302 in the corresponding embodiment are not repeated here.

[0119] In some optional implementations of this embodiment, the image processing device 600 further includes: a detection unit (not shown in the figure). The detection unit is configured to: detect whether the size of the image to be processed is a standard size; in response to the size of the image to be processed not being a standard size, adjust the image to be processed to the standard size.

[0120] In some optional implementations of the present embodiment, the above-mentioned image generation model includes: a graphic text recognition module, a large language model and a text generation image model, and the above-mentioned result obtaining unit 602 is further configured to: input the image to be processed into the graphic text recognition module to obtain the recognized text output by the graphic text recognition module; input the recognized text into the large language model to obtain the prompt word text output by the large language model; input the prompt word text into the text generation image model to obtain the generated image output by the text generation image model.

[0121] In some optional implementations of this embodiment, the device 600 further includes: a receiving unit (not shown in the figure) and a text input unit (not shown in the figure). The receiving unit may be configured to receive an image processing requirement text. The text input unit may be configured to, after obtaining the recognition text, splice the recognition text with the image processing requirement text to obtain second splicing information; input the second splicing information into the large language model to obtain a new prompt word text output by the large language model; input the new prompt word text into the text generation image model to obtain a new generated image output by the text generation image model.

[0122] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0123] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0124] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0125] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0126] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0127] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as an image generation model training method or an image processing method. For example, in some embodiments, the image generation model training method or the image processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the image generation model training method or the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the image generation model training method or the image processing method in any other suitable manner (eg, by means of firmware).

[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable image generation model training device, an image processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0133] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0135] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training an image generation model, the method comprising: Acquire an image sample set, wherein the image sample set includes at least one image sample; Obtain a pre-built image generation network, the image generation network comprising: a picture and text recognition module, a large language model and a text generation image model connected in sequence, the picture and text recognition module obtains a recognition text based on an input image; the large language model obtains a prompt word text with multi-image description features based on the recognition text; the text generation image model obtains a generated image based on the prompt word text; Inputting image samples selected from the image sample set into the image generation network to obtain a generated image output by the image generation network; scoring the generated image using an image scoring model to obtain an evaluation value of the generated image; calculating a network loss value of the image generation network based on the evaluation value; training the image generation network based on the network loss value of the image generation network to obtain a trained image generation model; Among them, the image scoring model is obtained by training a multimodal image and text recognition network, the input of the multimodal image and text recognition network includes a first image sample, and the output includes a first score, the first image sample includes a first image generated by the image generation model and a score of the first image obtained by manually scoring the first image, and the score of the first image, the first score and the loss function of the multimodal image and text recognition network are used to calculate the loss value of the multimodal image and text recognition network.

2. The method according to claim 1, wherein: The calculating the network loss value of the image generation network based on the evaluation value comprises: Obtaining a loss function of the text-to-image model; Calculating a model loss value of the text-generated image model based on the selected image sample and the loss function of the text-generated image model; Based on the evaluation value, the model loss value is adjusted to obtain the network loss value.

3. The method according to claim 1 or 2, wherein: The step of training the image generation network based on the network loss value of the image generation network to obtain a trained image generation model comprises: In response to a network loss value of the image generation network satisfying a training completion condition, the image generation network is used as an image generation model.

4. The method according to claim 3, wherein: The method further comprises: In response to a network loss value of the image generation network not satisfying a training completion condition, adjusting parameters of the large language model based on the network loss value, and continuing to train the image generation network.

5. The method according to claim 1, wherein: The multi-image description features include: element features, composition features and style features.

6. The method according to claim 1, wherein: The training steps of the multimodal image-text recognition network include: Perform the first training step: Inputting the first image sample and the obtained description text into the multimodal image-text recognition network to obtain an answer text output by the multimodal image-text recognition network; splicing the first image sample, the description text, the answer text and the rating text to obtain first splicing information, and inputting the first splicing information into the multimodal image-text recognition network to obtain the first rating; and obtaining a first rating model based on the loss value of the multimodal image-text recognition network; Perform the second training step: Inputting the acquired second image sample and the scoring text into the first scoring model to obtain a second score output by the first scoring model; Based on the second score, calculating the loss value of the first scoring model; An image scoring model is obtained based on the loss value of the first scoring model.

7. An image processing method, the method comprising: Get the image to be processed; The image to be processed is input into an image generation model generated by the method according to any one of claims 1 to 6 to obtain an image generation result of the image to be processed.

8. The method according to claim 7, before inputting the image to be processed into the image generation model, the method further comprises: Detecting whether the size of the image to be processed is a standard size; In response to the size of the image to be processed not being a standard size, the image to be processed is adjusted to a standard size.

9. The method according to claim 7, wherein the image generation model comprises: An image-text recognition module, a large language model, and a text-generated image model, wherein the inputting of the image to be processed into the image generation model generated by the method according to any one of claims 1 to 6, and outputting the image generation result of the image to be processed comprises: Inputting the image to be processed into the image-text recognition module to obtain the recognized text output by the image-text recognition module; Inputting the recognized text into the large language model to obtain a prompt word text output by the large language model; The prompt word text is input into the text-to-image model to obtain a generated image output by the text-to-image model.

10. The method according to claim 9, further comprising: Receiving an image processing requirement text; wherein the requirement text is a requirement for a generated image to be generated; After obtaining the recognition text, splicing the recognition text with the image processing requirement text to obtain second splicing information; Inputting the second concatenated information into the large language model to obtain a new prompt word text output by the large language model; The new prompt word text is input into the text-to-image model to obtain a new generated image output by the text-to-image model.

11. An image generation model training device, the device comprising: a set acquisition unit, configured to acquire an image sample set, wherein the image sample set includes at least one image sample; The network acquisition unit is configured to acquire a pre-built image generation network, wherein the image generation network includes: a picture and text recognition module, a large language model, and a text generation image model connected in sequence, wherein the picture and text recognition module obtains a recognition text based on an input image; the large language model obtains a prompt word text with multi-image description features based on the recognition text; and the text generation image model obtains a generated image based on the prompt word text; A sample input unit is configured to input an image sample selected from the image sample set into the image generation network to obtain a generated image output by the image generation network; a scoring unit, configured to score the generated image using an image scoring model to obtain an evaluation value of the generated image; A calculation unit, configured to calculate a network loss value of the image generation network based on the evaluation value; A model obtaining unit is configured to train the image generation network based on a network loss value of the image generation network to obtain a trained image generation model; Among them, the image scoring model is obtained by training a multimodal image and text recognition network, the input of the multimodal image and text recognition network includes a first image sample, and the output includes a first score, the first image sample includes a first image generated by the image generation model and a score of the first image obtained by manually scoring the first image, and the score of the first image, the first score and the loss function of the multimodal image and text recognition network are used to calculate the loss value of the multimodal image and text recognition network.

12. The device according to claim 11, wherein The calculation unit is further configured to: obtain the loss function of the text-generated image model; calculate the model loss value of the text-generated image model based on the selected image sample and the loss function of the text-generated image model; and adjust the model loss value based on the evaluation value to obtain the network loss value.

13. The device according to claim 11 or 12, wherein: The model obtaining unit is further configured to: in response to a network loss value of the image generation network satisfying a training completion condition, use the image generation network as an image generation model.

14. The device according to claim 13, wherein: The device also includes: an adjustment unit configured to adjust the parameters of the large language model and control the operation of the sample input unit based on the network loss value in response to the network loss value of the image generation network not satisfying the training completion condition.

15. The device according to claim 11, wherein The multi-image description features include: element features, composition features and style features.

16. The device according to claim 11, wherein The multimodal image and text recognition network is trained by a training unit, and the training unit is configured to: input the first image sample and the obtained description text into the multimodal image and text recognition network to obtain the answer text output by the multimodal image and text recognition network; splice the first image sample, the description text, the answer text and the scoring text to obtain first splicing information, and input the first splicing information into the multimodal image and text recognition network to obtain the first score; obtain a first scoring model based on the loss value of the multimodal image and text recognition network; input the obtained second image sample and the scoring text into the first scoring model to obtain a second score output by the first scoring model; calculate the loss value of the first scoring model based on the second score; and obtain an image scoring model based on the loss value of the first scoring model.

17. An image processing device, comprising: An image acquisition unit, configured to acquire an image to be processed; The result obtaining unit is configured to input the image to be processed into an image generation model generated by the device according to any one of claims 11 to 16, and output an image generation result of the image to be processed.

18. The apparatus according to claim 17, further comprising: A detection unit, wherein the detection unit is configured to: detect whether the size of the image to be processed is a standard size; In response to the size of the image to be processed not being a standard size, the image to be processed is adjusted to a standard size.

19. The device according to claim 17, wherein: The image generation model includes: a picture-text recognition module, a large language model and a text-to-image generation model, and the result obtaining unit is further configured to: input the image to be processed into the picture-text recognition module to obtain the recognized text output by the picture-text recognition module; input the recognized text into the large language model to obtain the prompt word text output by the large language model; input the prompt word text into the text-to-image generation model to obtain the generated image output by the text-to-image generation model.

20. The apparatus according to claim 19, further comprising: A receiving unit, configured to receive an image processing requirement text; The text input unit is configured to, after obtaining the recognition text, splice the recognition text with the image processing requirement text to obtain second splicing information; input the second splicing information into the large language model to obtain a new prompt word text output by the large language model; input the new prompt word text into the text generation image model to obtain a new generated image output by the text generation image model.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 10.

Images