Image generation method and device, equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of image generation, is applied to a scene for generating an image through a description text, and relates to an image generation method, device and equipment and a storage medium thereof. Predicting a text type of the text description data; if the text type is not the preset target text type, prompt content for modifying the text description data is sent to the target end; if the text type is the target text type, forwarding the text description data to a preset image generation model, and generating a to-be-classified image; and inputting all the to-be-classified images into the pre-trained image classification model, identifying the to-be-classified images of the target output type, and outputting all the to-be-classified images of the target output type to the target end. According to the image generation method provided by the invention, the text category prediction model, the image generation model and the image classification model are combined, so that prohibited contents in subsequently generated images are avoided, and the generated images are ensured to meet network supervision requirements.

Description

Technical Field

[0001] The present application relates to the technical field of image generation, and is applied to generating image scenarios with descriptive texts, and in particular to an image generation method, apparatus, device, and storage medium thereof. Background Art

[0002] Image generation is one of the research directions in the field of computer vision. Using the diffusion model to generate images is more diverse and of higher quality than GAN. The training process of the diffusion model is also more stable, and it has gradually developed into a mainstream method in the field of image generation. The diffusion model itself is a multimodal model, and multimodal image generation applications have been developed in a short period of time, including text generation to image, image conversion to image, text generation to video, etc.

[0003] Due to the high diversity of images generated based on the diffusion model and the variety of content, some images have compliance and security issues or hidden dangers, such as images with revealing clothing, private information, etc. At present, in order to solve such problems, most of them are generated offline and then manually verified to build a gallery. However, this solution is not only time-consuming and labor-intensive, but also difficult to supervise in the supervision of image generation. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an image generation method, device, equipment and storage medium thereof to solve the problem that the existing image generation based on descriptive text is not only time-consuming and labor-intensive, but also difficult to supervise in the supervision of image generation.

[0005] In order to solve the above technical problems, the embodiment of the present application provides an image generation method, which adopts the following technical solution:

[0006] A method for generating an image comprises the following steps:

[0007] Step 201, obtaining text description data for image generation;

[0008] Step 202, inputting the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model;

[0009] Step 203, if the text type is not a preset target text type, sending a prompt content for modifying the text description data to the target end;

[0010] Step 204: if the text type is the target text type, forwarding the text description data to a preset image generation model to obtain all images to be classified generated by the image generation model;

[0011] Step 205, inputting all the images to be classified into the pre-trained image classification model to identify images of the target output type;

[0012] Step 206: output images of all target output types to the target end.

[0013] Furthermore, before executing the step of inputting the text description data into the pre-trained text category prediction model to obtain the text type output by the text category prediction model, the method further includes:

[0014] Constructing a text training set containing positive and negative sample data, wherein the positive sample data refers to text description data that does not contain prohibited description text, and the negative sample data refers to text description data that contains prohibited description text;

[0015] Inputting the text training set into the text category prediction model to be trained, performing binary classification learning training, and training output nodes corresponding to positive sample data and negative sample data respectively;

[0016] Set the output node corresponding to the positive sample data to the output node corresponding to the target text type;

[0017] Set the output node corresponding to the negative sample data to the output node corresponding to the non-target text type;

[0018] Obtain a pre-trained text category prediction model;

[0019] The step of inputting the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model specifically includes:

[0020] Inputting the text description data into the pre-trained text category prediction model;

[0021] Predicting the output node corresponding to the text description data through the pre-trained text category prediction model;

[0022] The text type corresponding to the text description data is determined according to the output node corresponding to the text description data.

[0023] Furthermore, after executing the step of sending the prompt content for modifying the text description data to the target end, the method further includes:

[0024] Obtaining the modified text description data of the target end;

[0025] The modified text description data is re-input into the pre-trained text classification model.

[0026] Furthermore, the image generation model includes a text-to-image model based on a diffusion model. Before executing the step of forwarding the text description data to a preset image generation model to obtain all images to be classified generated by the image generation model, the method further includes:

[0027] Inputting the number of picture outputs required by the target end as a first configuration parameter in the image generation model;

[0028] Selecting a super generation mode as an optional generation mode of the image generation model, and setting a super generation parameter as a second configuration parameter, wherein the optional generation modes include a reduced generation mode, an equal generation mode, and a super generation mode;

[0029] The step of forwarding the text description data to a preset image generation model to obtain all images to be classified generated by the image generation model includes:

[0030] According to the preset calculation formula:

[0031] Q=|l1×l2|,

[0032] Generate Q images to be classified, where Q represents the number of all images to be classified, l1 represents the first configuration parameter, l2 represents the second configuration parameter, l1 is a positive integer, l2 is any multiple greater than 1, and |l1×l2| represents rounding up the result of multiplying l1 by l2.

[0033] Furthermore, before executing the step of inputting all the images to be classified into the pre-trained image classification model to identify images of the target output type, the method further includes:

[0034] Constructing an image training set including positive and negative sample images, wherein the positive sample images refer to images that do not contain prohibited content, and the negative sample images refer to images that contain prohibited content, wherein the prohibited content includes prohibited text and prohibited images;

[0035] Inputting the image training set into the image classification model to be trained, performing binary classification learning training, and training output nodes corresponding to the positive sample images and the negative sample images respectively;

[0036] The output node corresponding to the positive sample image is set to the output node corresponding to the image of the target output type;

[0037] The output node corresponding to the negative sample image is set to the output node corresponding to the image of the non-target output type;

[0038] Obtain a pre-trained image classification model;

[0039] The step of inputting all the images to be classified into the pre-trained image classification model to identify images of the target output type specifically includes:

[0040] Inputting all the images to be classified into the pre-trained image classification model;

[0041] Predicting the output nodes corresponding to all the images to be classified respectively through the pre-trained image classification model;

[0042] According to the output nodes respectively corresponding to all the images to be classified, images of a target output type are identified from all the images to be classified.

[0043] Furthermore, before executing the step of outputting images of all target output types to the target end, the method further includes:

[0044] Step 401, counting the number of images of all target output types, recorded as the number of images to be output;

[0045] Step 402, comparing the relationship between the number of images to be output and the first configuration parameter;

[0046] Step 403: if the number of images to be output is equal to the first configuration parameter, output all images of the target output type to the target end;

[0047] Step 404: if the number of images to be output is greater than the first configuration parameter, an image resolution screening method is used to screen out images with the same optimal resolution as the number of the first configuration parameter from the number of images to be output, and update them as images to be output;

[0048] Step 405: If the number of images to be output is less than the first configuration parameter, incremental update processing is performed on the images of all target output types according to a preset incremental update method, and after the incremental update processing, steps 401 to 402 are re-executed until the number of images to be output is not less than the first configuration parameter, and the incremental update processing is stopped.

[0049] Furthermore, the step of performing incremental update processing on the images of all target output types according to the preset incremental update mode specifically includes:

[0050] Calculating a difference between the number of images to be output and the first configuration parameter;

[0051] Inputting the difference in the image generation model as the number of picture outputs when performing incremental update;

[0052] Replacing the first configuration parameter with the difference, and continuing to execute step 204 to step 205 to obtain an image of the target output type newly identified;

[0053] The image of the target output type that is most recently identified is cached together with images of the target output type that are previously identified, and all cached images are used as images of all target output types obtained after incremental update processing.

[0054] Further, after executing the step of outputting images of all target output types to the target end, the method further includes:

[0055] Acquire the batch of non-preferred images returned by the target end;

[0056] Extracting common image features from the batch of non-preferred images;

[0057] The common image features are used as supplementary classification features of the output nodes corresponding to the negative sample images to update the image classification model.

[0058] In order to solve the above technical problems, the embodiment of the present application also provides an image generating device, which adopts the following technical solution:

[0059] An image generating device, comprising:

[0060] A text description data acquisition module, used to acquire text description data for image generation;

[0061] A text type prediction module, used for inputting the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model;

[0062] A first branch processing module, configured to send a prompt content for modifying the text description data to a target end if the text type is not a preset target text type;

[0063] A second branch processing module is used for forwarding the text description data to a preset image generation model to obtain all images to be classified generated by the image generation model if the text type is the target text type;

[0064] A target image recognition module is used to input all the images to be classified into a pre-trained image classification model to identify images of a target output type;

[0065] The target image output module is used to output images of all target output types to the target end.

[0066] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0067] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned image generation method when executing the computer-readable instructions.

[0068] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0069] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the image generation method described above.

[0070] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0071] The image generation method described in the embodiment of the present application obtains text description data for image generation provided by the user; inputs the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model; if the text type is not the preset target text type, a prompt content for modifying the text description data is sent to the target end; if the text type is the target text type, the text description data is forwarded to the preset image generation model to obtain all images to be classified generated by the image generation model; all images to be classified are input into a pre-trained image classification model to identify the images to be classified of the target output type and output all images to be classified of the target output type to the target end. The image generation method described in the present application combines a text category prediction model, an image generation model, and an image classification model to avoid the presence of prohibited content in subsequently generated images and ensure that the generated images meet network regulatory requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0073] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0074] Figure 2 is a flow chart of an embodiment of an image generation method according to the present application;

[0075] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;

[0076] Figure 4 yes Figure 2 A flowchart of a specific embodiment of adjusting the output image before step 206 is shown;

[0077] Figure 5 yes Figure 4 A flowchart of a specific embodiment of performing incremental update processing on images of all target output types according to a preset incremental update method in step 405;

[0078] Figure 6 is a schematic structural diagram of an embodiment of an image generating device according to the present application;

[0079] Figure 7 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0080] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0081] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0082] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0083] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0084] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0085] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0086] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0087] It should be noted that the image generating method provided in the embodiment of the present application is generally executed by a terminal device, and accordingly, the image generating device is generally arranged in the terminal device.

[0088] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0089] Continue to refer Figure 2 , shows a flow chart of an embodiment of an image generation method according to the present application. The image generation method comprises the following steps:

[0090] Step 201, obtaining text description data for image generation;

[0091] Specifically, the text description data used for image generation, for example, the lesion description text in a medical test report, can be used to reversely construct a lesion simulation image based on the lesion description text to assist patients and doctors in understanding their condition; for another example, the generation of holiday greeting cards in financial tips can be used to generate corresponding holiday greeting cards using the card generation requirements in the text description data, so as to achieve regular customer greetings and assist financial institutions in maintaining customer relationships.

[0092] In this embodiment, the text description data for image generation may differ depending on the different providing ends. For example, if the providing end is a business service end, the text description data uploaded or pushed by the business service end may be obtained directly through a passive receiving method. Furthermore, if the providing end is each user end in a collection scenario, the text description data filled in by each user end may be obtained through an active crawling method.

[0093] Step 202, inputting the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model;

[0094] In this embodiment, the purpose of learning and training the text category prediction model is to distinguish text description data containing prohibited descriptive text and text description data not containing prohibited descriptive text, so as to avoid the presence of prohibited content in subsequently generated images and ensure that the generated images meet network supervision requirements. The prohibited descriptive text includes personal privacy information, such as identity card number information, bank card account information, etc.

[0095] Correspondingly, the text category prediction model can be trained by the encoder part and the softmax function normalization part in the transformer neural network. The transformer is a sequence model based on the attention mechanism. Since the text description data is classified, the encoder part in the transformer is used, combined with the multi-head self-attention layer and the feedforward neural network between the layers in the encoder. It is sufficient to learn the text features corresponding to the positive and negative sample data respectively when the positive and negative sample data are sufficient, and set different output nodes for the text types corresponding to the positive and negative sample data according to the text features.

[0096] Step 203, if the text type is not a preset target text type, sending a prompt content for modifying the text description data to the target end;

[0097] Specifically, text description data that does not contain prohibited description text is defined in advance as data corresponding to the target text type. At this time, the text type is not the preset target text type, which can be understood as the presence of prohibited data in the text description data used for image generation. In this case, a prompt content for modifying the text description data is sent to the target end, so that before the image is generated with the text description data, a prohibited filtering is performed based on the text content, and the prohibited text data is screened out early, thereby avoiding blindly using the text description data for image generation, and correspondingly reducing the resource consumption of invalid conversion results when converting text to image. The target end is different according to the different acquisition ends of the text description data used for image generation.

[0098] Step 204: if the text type is the target text type, forwarding the text description data to a preset image generation model to obtain all images to be classified generated by the image generation model;

[0099] Specifically, if the text type is the target text type, that is, it is identified that the text description data does not contain prohibited data, the text description data is forwarded to a preset image generation model to obtain all images to be classified generated by the image generation model, thereby ensuring that the generated images do not contain prohibited content as much as possible.

[0100] Correspondingly, the preset image generation model may be a multimodal processing model based on a Diffusion Model, which can generate images according to text description data.

[0101] Step 205, inputting all the images to be classified into the pre-trained image classification model to identify images of the target output type;

[0102] Specifically, by performing image classification and recognition again on the images generated by the image generation model, non-target output type images can be avoided in the generated images. The target output type can be understood as a pre-set image that meets the network regulatory output requirements, and the non-target output type can be understood as an image that does not meet the network regulatory output requirements, thereby further ensuring the compliance of the output image.

[0103] Correspondingly, the image classification model can be formed by learning and training based on the resnet-50 deep residual network combined with the softmax function normalization part. The ResNet residual network significantly improves the accuracy of image classification through its unique residual block design and jump connection mechanism.

[0104] Step 206: output images of all target output types to the target end.

[0105] In this embodiment, the text description data for image generation provided by the user is obtained; the text description data is input into a pre-trained text category prediction model to obtain the text type output by the text category prediction model; if the text type is not the preset target text type, a prompt content for modifying the text description data is sent to the target end; if the text type is the target text type, the text description data is forwarded to a preset image generation model to obtain all images to be classified generated by the image generation model; all images to be classified are input into a pre-trained image classification model to identify the images to be classified of the target output type and output all images to be classified of the target output type to the target end. The image generation method described in the present application combines a text category prediction model, an image generation model and an image classification model to avoid the presence of prohibited content in subsequently generated images and ensure that the generated images meet network regulatory requirements.

[0106] In some optional implementations, before step 202, a step of learning and training the text category prediction model is also included, and the learning and training of the text category prediction model includes the following steps: constructing a text training set containing positive and negative sample data, wherein the positive sample data refers to text description data that does not contain prohibited description text, and the negative sample data refers to text description data that contains prohibited description text; inputting the text training set into the text category prediction model to be trained, performing binary classification learning and training, and training output nodes corresponding to the positive sample data and the negative sample data respectively; setting the output node corresponding to the positive sample data as the output node corresponding to the target text type; setting the output node corresponding to the negative sample data as the output node corresponding to the non-target text type; and obtaining a pre-trained text category prediction model.

[0107] Correspondingly, by learning and training the text category prediction model, the pre-trained text category prediction model can identify the type of text description data, ensuring that the text description data for subsequent image generation does not contain prohibited content, which complies with network supervision requirements.

[0108] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 202 shown includes the following steps:

[0109] Step 301, inputting the text description data into the pre-trained text category prediction model;

[0110] Step 302, predicting the output node corresponding to the text description data through the pre-trained text category prediction model;

[0111] Specifically, the output node can also be understood as a classification node, and different output nodes correspond to different text types.

[0112] Step 303: Determine the text type corresponding to the text description data according to the output node corresponding to the text description data.

[0113] In this embodiment, after executing the step of sending a prompt content for modifying the text description data to the target end, the method also includes: obtaining the modified text description data of the target end; and re-inputting the modified text description data into the pre-trained text classification model.

[0114] Accordingly, after discovering that the text description data contains prohibited content, a modification prompt is promptly sent to the target end, and the modified text description data is promptly obtained, thereby avoiding long-term suspension of task processing.

[0115] In this embodiment, the image generation model includes a text-to-image model based on a diffusion model. In some optional implementations, before forwarding the text description data to a preset image generation model in step 204 and obtaining all the images to be classified generated by the image generation model, the following steps are also included: inputting the number of image outputs required by the target end as a first configuration parameter in the image generation model; selecting an over-generation mode as an optional generation mode of the image generation model, and setting an over-generation parameter as a second configuration parameter, wherein the optional generation modes include a reduced generation mode, an equal generation mode, and an over-generation mode;

[0116] Specifically, the reduced-volume generation mode, equal-volume generation mode, and super-volume generation mode can be understood as the generation modes required in different generation scenarios. The reduced-volume generation mode, for example, inputs multiple text description data and only generates a small number of images. It is often used in the precise image generation scenario, and the multiple text description data are used to describe one image; the equal-volume generation mode, for example, inputs 10 text description data to generate 10 images. It is often used in the normal image generation scenario, and one text description data corresponds to one generated image; and the super-volume generation mode is often used in the scenario of batch generation of images from a small amount of text description data, for example, outputting 10 text description data to generate 100 images; the specific usage scenarios can be adjusted according to business needs, and will not be repeated here. Different generation modes are provided for selection, which are more user-friendly and closer to actual business needs.

[0117] In some optional implementations, step 204 forwards the text description data to a preset image generation model, and the step of obtaining all images to be classified generated by the image generation model includes: according to a preset calculation formula:

[0118] Q=|l1×l2|,

[0119] Generate Q images to be classified, where Q represents the number of all images to be classified, l1 represents the first configuration parameter, l2 represents the second configuration parameter, l1 is a positive integer, l2 is any multiple greater than 1, and |l1×l2| represents rounding up the result of multiplying l1 by l2.

[0120] The over-generation mode is adopted to ensure that the final output image meets the output quantity requirements of the target end.

[0121] In some optional implementations, before step 205, the method further includes learning and training the image classification model, including the following steps: constructing an image training set containing positive and negative sample images, wherein the positive sample images refer to images that do not contain prohibited content, and the negative sample images refer to images that contain prohibited content, and the prohibited content includes prohibited text and prohibited pictures; inputting the image training set into the image classification model to be trained, performing binary classification learning training, and training the output nodes corresponding to the positive sample images and the negative sample images, respectively; setting the output nodes corresponding to the positive sample images to the output nodes corresponding to the images of the target output type; setting the output nodes corresponding to the negative sample images to the output nodes corresponding to the images of the non-target output type; and obtaining the pre-trained image classification model.

[0122] By learning and training the image classification model, the pre-trained image classification model can perform image classification and recognition on the images generated by the image generation model again. When non-target output type images exist in the images generated by the image generation model, this type of image is prevented from being output to the target end.

[0123] In some optional implementations, step 205 includes the following steps: inputting all the images to be classified into the pre-trained image classification model; predicting the output nodes corresponding to all the images to be classified through the pre-trained image classification model; and identifying images of the target output type from all the images to be classified according to the output nodes corresponding to all the images to be classified.

[0124] By performing image classification and recognition again on the images generated by the image generation model, non-target output type images can be avoided in the generated images. The target output type can be understood as a pre-set image that meets the network regulatory output requirements, and the non-target output type can be understood as an image that does not meet the network regulatory output requirements, thereby further ensuring the compliance of the output image.

[0125] Continue to refer Figure 4 , Figure 4 yes Figure 2The flowchart of a specific embodiment of adjusting the output image before step 206 is shown. In some optional implementations, adjusting the image before output also includes the following steps:

[0126] Step 401, counting the number of images of all target output types, recorded as the number of images to be output;

[0127] Step 402, comparing the relationship between the number of images to be output and the first configuration parameter;

[0128] Step 403: if the number of images to be output is equal to the first configuration parameter, output all images of the target output type to the target end;

[0129] Step 404: if the number of images to be output is greater than the first configuration parameter, an image resolution screening method is used to screen out images with the same optimal resolution as the number of the first configuration parameter from the number of images to be output, and update them as images to be output;

[0130] Step 405: If the number of images to be output is less than the first configuration parameter, incremental update processing is performed on the images of all target output types according to a preset incremental update method, and after the incremental update processing, steps 401 to 402 are re-executed until the number of images to be output is not less than the first configuration parameter, and the incremental update processing is stopped.

[0131] By counting the number of images of all target output types before outputting them to the target end, and comparing the number with the first configuration parameter, it is ensured that the images output to the target end meet the output quantity requirement.

[0132] Continue to refer Figure 5 , Figure 5 yes Figure 4 The flowchart of a specific embodiment of performing incremental update processing on the images of all target output types according to the preset incremental update method in step 405 is shown. In some optional implementations, the incremental update processing further includes the following steps:

[0133] Step 501, calculating the difference between the number of images to be output and the first configuration parameter;

[0134] Step 502, inputting the difference value into the image generation model as the number of picture outputs when performing incremental update;

[0135] Step 503, replacing the first configuration parameter with the difference, and continuing to execute steps 204 to 205 to obtain an image of the target output type that is newly identified;

[0136] Specifically, the first configuration parameter is replaced by the difference, and step 204 to step 205 are continued to be executed to obtain the image of the latest identified target output type, thereby realizing dynamic incremental update according to the missing amount of output quantity, which is more scientific and avoids blind update and causing a large amount of computer resource waste.

[0137] Step 504: cache the newly identified target output type image together with previously identified target output type images, and use all cached images as images of all target output types obtained after incremental update processing.

[0138] By adopting a dynamic incremental update method, it is ensured that the number of images output to the target end meets the output quantity requirement.

[0139] In this embodiment, after executing the step of outputting all images of the target output type to the target end, the method also includes: obtaining a batch of non-preferred images returned by the target end; extracting common image features from the batch of non-preferred images; and using the common image features as supplementary classification features of the output nodes corresponding to the negative sample images to update the image classification model.

[0140] Specifically, the batch of non-preferred images returned by the target end is obtained, common image features are extracted from the batch of non-preferred images, the common image features are used as supplementary classification features of the output nodes corresponding to the negative sample images, and the image classification model is updated. This facilitates further control processing of the output image in combination with the preferences of the target end, and the generation control during the generation under normal circumstances is post-processed and changed to output control, which reduces the processing difficulty.

[0141] This application obtains text description data for image generation provided by the user; inputs the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model; if the text type is not the preset target text type, a prompt content for modifying the text description data is sent to the target end; if the text type is the target text type, the text description data is forwarded to a preset image generation model to obtain all images to be classified generated by the image generation model; all images to be classified are input into a pre-trained image classification model to identify the images to be classified of the target output type and output all images to be classified of the target output type to the target end. The image generation method described in this application combines a text category prediction model, an image generation model, and an image classification model to avoid the presence of prohibited content in subsequently generated images and ensure that the generated images meet network regulatory requirements.

[0142] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0143] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0144] In an embodiment of the present application, by obtaining text description data for image generation provided by a user; inputting the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model; if the text type is not the preset target text type, sending a prompt to modify the text description data to the target end; if the text type is the target text type, forwarding the text description data to a preset image generation model to obtain all images to be classified generated by the image generation model; inputting all images to be classified into a pre-trained image classification model to identify images to be classified of the target output type and outputting all images to be classified of the target output type to the target end. The image generation method described in the present application combines a text category prediction model, an image generation model, and an image classification model to avoid the presence of prohibited content in subsequently generated images and ensure that the generated images meet network regulatory requirements.

[0145] Further references Figure 6 , as a response to the above Figure 2 The present application provides an embodiment of an image generating device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0146] like Figure 6 As shown, the image generation device 600 described in this embodiment includes: a text description data acquisition module 601, a text type prediction module 602, a first branch processing module 603, a second branch processing module 604, a target image recognition module 605 and a target image output module 606. Among them:

[0147] A text description data acquisition module 601 is used to acquire text description data for image generation;

[0148] A text type prediction module 602 is used to input the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model;

[0149] The first branch processing module 603 is used to send a prompt content for modifying the text description data to the target end if the text type is not a preset target text type;

[0150] The second branch processing module 604 is used for forwarding the text description data to a preset image generation model to obtain all images to be classified generated by the image generation model if the text type is the target text type;

[0151] A target image recognition module 605 is used to input all the images to be classified into a pre-trained image classification model to identify images of a target output type;

[0152] The target image output module 606 is used to output images of all target output types to the target end.

[0153] This application obtains text description data for image generation provided by the user; inputs the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model; if the text type is not the preset target text type, a prompt content for modifying the text description data is sent to the target end; if the text type is the target text type, the text description data is forwarded to a preset image generation model to obtain all images to be classified generated by the image generation model; all images to be classified are input into a pre-trained image classification model to identify the images to be classified of the target output type and output all images to be classified of the target output type to the target end. The image generation method described in this application combines a text category prediction model, an image generation model, and an image classification model to avoid the presence of prohibited content in subsequently generated images and ensure that the generated images meet network regulatory requirements.

[0154] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0155] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0156] To solve the above technical problems, the present application also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0157] The computer device 7 includes a memory 7a, a processor 7b, and a network interface 7c which are interconnected and communicated via a system bus. It should be noted that: Figure 7 Only a computer device 7 having a component memory 7a, a processor 7b, and a network interface 7c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0158] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0159] The memory 7a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 7a can be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, the memory 7a can also be an external storage device of the computer device 7, such as a plug-in hard disk equipped on the computer device 7, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 7a can also include both the internal storage unit of the computer device 7 and its external storage device. In this embodiment, the memory 7a is generally used to store the operating system and various application software installed on the computer device 7, such as a computer-readable instruction of an image generation method, etc. In addition, the memory 7a can also be used to temporarily store various types of data that have been output or are to be output.

[0160] The processor 7b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 7b is generally used to control the overall operation of the computer device 7. In this embodiment, the processor 7b is used to run computer-readable instructions or process data stored in the memory 7a, such as computer-readable instructions for running the image generation method.

[0161] The network interface 7c may include a wireless network interface or a wired network interface. The network interface 7c is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0162] The computer device proposed in this embodiment belongs to the field of image generation technology and is applied to the scene of generating images with descriptive text. This application obtains the text description data provided by the user for image generation; inputs the text description data into the pre-trained text category prediction model to obtain the text type output by the text category prediction model; if the text type is not the preset target text type, a prompt content for modifying the text description data is sent to the target end; if the text type is the target text type, the text description data is forwarded to the preset image generation model to obtain all images to be classified generated by the image generation model; all images to be classified are input into the pre-trained image classification model, the images to be classified of the target output type are identified, and all images to be classified of the target output type are output to the target end. The image generation method described in this application combines the text category prediction model, the image generation model and the image classification model to avoid the existence of prohibited content in the subsequently generated images and ensure that the generated images meet network supervision requirements.

[0163] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the image generation method as described above.

[0164] The computer-readable storage medium proposed in this embodiment belongs to the field of image generation technology and is applied to the scene of generating images with descriptive text. This application obtains the text description data provided by the user for image generation; inputs the text description data into the pre-trained text category prediction model to obtain the text type output by the text category prediction model; if the text type is not the preset target text type, a prompt content for modifying the text description data is sent to the target end; if the text type is the target text type, the text description data is forwarded to the preset image generation model to obtain all images to be classified generated by the image generation model; all images to be classified are input into the pre-trained image classification model, and the images to be classified of the target output type are identified and all images to be classified of the target output type are output to the target end. The image generation method described in this application combines the text category prediction model, the image generation model and the image classification model to avoid the existence of prohibited content in the subsequently generated images and ensure that the generated images meet network supervision requirements.

[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0166] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An image generation method, characterized in that: The steps include: Step 201, obtaining text description data for image generation; Step 202, inputting the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model; Step 203, if the text type is not a preset target text type, sending a prompt content for modifying the text description data to the target end; Step 204: if the text type is the target text type, forwarding the text description data to a preset image generation model to obtain all images to be classified generated by the image generation model; Step 205, inputting all the images to be classified into the pre-trained image classification model to identify images of the target output type; Step 206: output images of all target output types to the target end.

2. The image generation method according to claim 1, characterized in that: Before executing the step of inputting the text description data into the pre-trained text category prediction model to obtain the text type output by the text category prediction model, the method further includes: Constructing a text training set containing positive and negative sample data, wherein the positive sample data refers to text description data that does not contain prohibited description text, and the negative sample data refers to text description data that contains prohibited description text; Inputting the text training set into the text category prediction model to be trained, performing binary classification learning training, and training output nodes corresponding to positive sample data and negative sample data respectively; Set the output node corresponding to the positive sample data to the output node corresponding to the target text type; Set the output node corresponding to the negative sample data to the output node corresponding to the non-target text type; Obtain a pre-trained text category prediction model; The step of inputting the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model specifically includes: Inputting the text description data into the pre-trained text category prediction model; Predicting the output node corresponding to the text description data through the pre-trained text category prediction model; The text type corresponding to the text description data is determined according to the output node corresponding to the text description data.

3. The image generation method according to claim 1, characterized in that: After executing the step of sending the prompt content for modifying the text description data to the target end, the method further includes: Obtaining the modified text description data of the target end; The modified text description data is re-input into the pre-trained text classification model.

4. The image generation method according to claim 1, characterized in that: The image generation model includes a text-to-image model based on a diffusion model. Before executing the step of forwarding the text description data to a preset image generation model to obtain all images to be classified generated by the image generation model, the method further includes: Inputting the number of picture outputs required by the target end as a first configuration parameter in the image generation model; Selecting a super generation mode as an optional generation mode of the image generation model, and setting a super generation parameter as a second configuration parameter, wherein the optional generation modes include a reduced generation mode, an equal generation mode, and a super generation mode; The step of forwarding the text description data to a preset image generation model to obtain all images to be classified generated by the image generation model includes: According to the preset calculation formula: Q＝|l1×l2|， Generate Q images to be classified, where Q represents the number of all images to be classified, l1 represents the first configuration parameter, l2 represents the second configuration parameter, l1 is a positive integer, l2 is any multiple greater than 1, and |l1×l2| represents rounding up the result of multiplying l1 by l2.

5. The image generation method according to claim 1, characterized in that: Before executing the step of inputting all the images to be classified into the pre-trained image classification model to identify images of the target output type, the method further includes: Constructing an image training set including positive and negative sample images, wherein the positive sample images refer to images that do not contain prohibited content, and the negative sample images refer to images that contain prohibited content, wherein the prohibited content includes prohibited text and prohibited images; Inputting the image training set into the image classification model to be trained, performing binary classification learning training, and training output nodes corresponding to the positive sample images and the negative sample images respectively; The output node corresponding to the positive sample image is set to the output node corresponding to the image of the target output type; The output node corresponding to the negative sample image is set to the output node corresponding to the image of the non-target output type; Obtain a pre-trained image classification model; The step of inputting all the images to be classified into the pre-trained image classification model to identify images of the target output type specifically includes: Inputting all the images to be classified into the pre-trained image classification model; Predicting the output nodes corresponding to all the images to be classified respectively through the pre-trained image classification model; According to the output nodes respectively corresponding to all the images to be classified, images of a target output type are identified from all the images to be classified.

6. The image generation method according to claim 4, characterized in that: Before executing the step of outputting all images of target output types to the target end, the method further includes: Step 401, counting the number of images of all target output types, recorded as the number of images to be output; Step 402, comparing the relationship between the number of images to be output and the first configuration parameter; Step 403: if the number of images to be output is equal to the first configuration parameter, output all images of the target output type to the target end; Step 404: if the number of images to be output is greater than the first configuration parameter, an image resolution screening method is used to screen out images with the same optimal resolution as the number of the first configuration parameter from the number of images to be output, and update them as images to be output; Step 405: If the number of images to be output is less than the first configuration parameter, incremental update processing is performed on the images of all target output types according to a preset incremental update method, and after the incremental update processing, steps 401 to 402 are re-executed until the number of images to be output is not less than the first configuration parameter, and the incremental update processing is stopped.

7. The image generation method according to claim 6, characterized in that: The step of performing incremental update processing on the images of all target output types according to the preset incremental update mode specifically includes: Calculating a difference between the number of images to be output and the first configuration parameter; Inputting the difference in the image generation model as the number of picture outputs when performing incremental update; Replacing the first configuration parameter with the difference, and continuing to execute step 204 to step 205 to obtain an image of the target output type newly identified; The image of the target output type that is most recently identified is cached together with images of the target output type that are previously identified, and all cached images are used as images of all target output types obtained after incremental update processing.

8. The image generation method according to claim 5, characterized in that: After executing the step of outputting all images of target output types to the target end, the method further includes: Acquire the batch of non-preferred images returned by the target end; Extracting common image features from the batch of non-preferred images; The common image features are used as supplementary classification features of the output nodes corresponding to the negative sample images to update the image classification model.

9. An image generating device, characterized in that: include: A text description data acquisition module, used to acquire text description data for image generation; A text type prediction module, used for inputting the text description data into a pre-trained text category prediction model to obtain the text type output by the text category prediction model; A first branch processing module, configured to send a prompt content for modifying the text description data to a target end if the text type is not a preset target text type; A second branch processing module is used for forwarding the text description data to a preset image generation model to obtain all images to be classified generated by the image generation model if the text type is the target text type; A target image recognition module is used to input all the images to be classified into a pre-trained image classification model to identify images of a target output type; The target image output module is used to output images of all target output types to the target end.

10. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the image generation method according to any one of claims 1 to 8 when executing the computer-readable instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image generation method according to any one of claims 1 to 8 are implemented.

Images