Style image generation model training method and device, electronic equipment and storage medium

Abstract

The invention provides a style image generation model training method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring an original image set, performing image content identification on the original image to obtain a style-free image content identification text, generating a style-free generation image set based on the style-free image content identification text, determining a style image content text based on a target style keyword and the style-free image content identification text, and displaying the style image content text. Determining a training text and image supervision information in a styleless training sample based on a styleless image content identification text and a styleless generation image, and determining a training text and image supervision information in a style training sample based on a style image content text and the original image, and training the to-be-trained target style image generation model based on the at least one non-style training sample and the at least one style training sample to obtain the trained target style image generation model, so that the image content and the image style can be better decoupled.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a style image generation model training method, device, electronic device, and storage medium. Background Art

[0002] The style image generation model is used to learn the style of an image and generate an image of that style. Specifically, text containing the image style is input into the style image generation model, and a style image corresponding to the style is output.

[0003] However, existing style image generation models tend to learn image content, making it difficult to decouple image content and image style, resulting in the image output by the style image generation model may not be an image corresponding to the style. Summary of the invention

[0004] The embodiments of the present disclosure provide a style image generation model training method, device, electronic device and storage medium.

[0005] In a first aspect, the present disclosure provides a style image generation model training method, comprising:

[0006] Obtaining an original image set, wherein the original images conform to a target style indicated by a target style keyword;

[0007] For an original image in the original image set, the following training sample generation operations are performed: image content recognition is performed on the original image to obtain a styleless image content recognition text; based on the styleless image content recognition text, a styleless generated image set is generated; based on the target style keyword and the styleless image content recognition text, the style image content text is determined; based on the styleless image content recognition text and the styleless generated image, the training text and image supervision information in the styleless training sample are determined, and based on the style image content text and the original image, the training text and image supervision information in the style training sample are determined;

[0008] Based on at least one non-style training sample and at least one style training sample, a target style image generation model to be trained is trained to obtain a trained target style image generation model.

[0009] In some optional implementations, determining the style image content text based on the target style keyword and the non-style image content recognition text includes:

[0010] The target style keywords and the unstyled image content recognition text are concatenated to obtain the style image content text.

[0011] In some optional implementations, recognizing text based on the content of the styleless image and generating a set of styleless generated images includes:

[0012] The styleless image content recognition text is input into the styleless image generation model to obtain a set of styleless generated images.

[0013] In some optional implementations, based on at least one non-style training sample and at least one style training sample, a target style image generation model to be trained is trained to obtain a trained target style image generation model, including:

[0014] For at least one non-style training sample and at least one style training sample, the following model parameter adjustment operation is performed: inputting the training text in the at least one non-style training sample and the at least one style training sample into the target style image generation model to be trained to obtain an actual generated image; adjusting the model parameters of the target style image generation model to be trained based on the difference between the actual generated image and the image supervision information in the at least one non-style training sample and the at least one style training sample;

[0015] The target style image generation model to be trained after the model parameters are adjusted is determined as the trained target style image generation model.

[0016] In some optional implementations, based on at least one non-style training sample and at least one style training sample, a target style image generation model to be trained is trained to obtain a trained target style image generation model, including:

[0017] For the training samples in at least one non-style training sample and at least one style training sample, the following model parameter adjustment operation is performed: inputting the training texts in the at least one non-style training sample and the at least one style training sample into the target style image generation model to be trained to obtain an actual generated image; adjusting the model parameters of the target style image generation model to be trained based on the difference between the image supervision information in the at least one non-style training sample and the at least one style training sample;

[0018] The target style text in the target style text set is input into the target style image generation model after the model parameters are adjusted to obtain the trained target style image generation model.

[0019] In some optional implementations, inputting the target style text in the target style text set into the target style image generation model after the model parameters are adjusted to obtain the trained target style image generation model, including:

[0020] Inputting the target style text in the target style text set into the target style image generation model after the model parameters are adjusted to obtain a target style generated image set;

[0021] Determine whether the proportion of images that meet the target style in the target style generated image set is greater than or equal to a preset proportion threshold;

[0022] In response to determining yes, determining the target style image generation model after parameter adjustment as the trained target style image generation model;

[0023] In response to a negative determination, the images in the target style generated image set that conform to the target style are added to the original image set, and the target style image generation model with adjusted parameters is determined as the target style image generation model to be trained, and the style image generation model training method is continued.

[0024] In some optional implementations, performing image content recognition on the original image to obtain styleless image content recognition text includes:

[0025] The original image is input into a pre-trained image content recognition model to obtain styleless image content recognition text.

[0026] In a second aspect, the present disclosure provides a style image generation method, comprising:

[0027] Obtaining image content description text and style description text of the image to be generated, wherein the style description text is associated with a target style keyword;

[0028] Based on the image content description text and the style description text, an image corresponding to the image content description text and the style description text is generated through a target style image generation model corresponding to the target style keywords, wherein the target style image generation model is pre-trained using any of the methods in the first aspect of the present disclosure.

[0029] In a third aspect, the present disclosure provides a style image generation model training device, comprising:

[0030] An original image set acquisition unit, used to acquire an original image set, wherein the original images conform to the target style indicated by the target style keyword;

[0031] The training sample generating unit is used to perform the following training sample generating operations on the original image in the original image set: performing image content recognition on the original image to obtain a styleless image content recognition text; generating a styleless generated image set based on the styleless image content recognition text; determining a style image content text based on a target style keyword and the styleless image content recognition text; determining training text and image supervision information in a styleless training sample based on the styleless image content recognition text and the styleless generated image, and determining training text and image supervision information in a style training sample based on the style image content text and the original image;

[0032] The target style image generation model training unit is used to train the target style image generation model to be trained based on at least one non-style training sample and at least one style training sample to obtain a trained target style image generation model.

[0033] In a fourth aspect, the present disclosure provides a style image generation device, comprising:

[0034] An image content description text and style description text acquisition unit, used to acquire the image content description text and style description text of the image to be generated, wherein the style description text is associated with a target style keyword;

[0035] An image generation unit is used to generate an image corresponding to the image content description text and the style description text based on the image content description text and the style description text through a target style image generation model corresponding to the target style keyword, wherein the target style image generation model is pre-trained using any method as described in the first aspect of the present disclosure.

[0036] In a fifth aspect, the present disclosure provides an electronic device, including:

[0037] one or more processors;

[0038] a storage device having one or more programs stored thereon,

[0039] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any embodiment of the first aspect and / or the second aspect of the present disclosure.

[0040] In a sixth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by one or more processors, the method described in any embodiment of the first aspect and / or second aspect of the present disclosure is implemented.

[0041] The style image generation model training method, device, electronic device and storage medium provided by the embodiments of the present disclosure first obtain a set of original images, wherein the original images conform to the target style indicated by the target style keyword. Then, for the original images in the original image set, the following training sample generation operations are performed: image content recognition is performed on the original image to obtain a styleless image content recognition text, a styleless generated image set is generated based on the styleless image content recognition text, a style image content text is determined based on the target style keyword and the styleless image content recognition text, training text and image supervision information in the styleless training sample are determined based on the styleless image content recognition text and the styleless generated image, and training text and image supervision information in the styleless training sample are generated based on the style image content text and the original image. Finally, based on at least one styleless training sample and at least one style training sample, the target style image generation model to be trained is trained to obtain a trained target style image generation model. That is, the target style image generation model to be trained is trained by using the non-style training samples and the style training samples, and then the target style image generation model is obtained by comparative learning of the non-style training samples and the style training samples, so that when the target style image generation model learns the style of the original image, the image content and the image style can be better decoupled, thereby improving the effect of the target style image generation model in generating stylized images. In addition, the present disclosure can achieve the above-mentioned effect with a small number of training image samples, thereby reducing the training cost of the target style image generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the drawings:

[0043] Figure 1 is a system architecture diagram to which an embodiment of the style image generation model training method disclosed herein may be applied;

[0044] Figure 2A is a flowchart of an embodiment of a style image generation model training method according to the present disclosure;

[0045] Figure 2B is a flowchart of an embodiment of a training sample generation operation according to the present disclosure;

[0046] Figure 2C is a decomposed flow chart of one embodiment of step 203 according to the present disclosure;

[0047] Figure 2D is a decomposed flow chart of yet another embodiment of step 203 according to the present disclosure;

[0048] Figure 3 is a flowchart of an embodiment of a style image generation method according to the present disclosure;

[0049] Figure 4 is a structural schematic diagram of an embodiment of a style image generation model training device according to the present disclosure;

[0050] Figure 5 is a schematic structural diagram of an embodiment of a style image generating device according to the present disclosure;

[0051] Figure 6 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0053] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0054] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the style image generation model training method, apparatus, terminal device, and storage medium of the present disclosure can be applied.

[0055] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0056] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as style image generation model training applications, style image generation applications, video conferencing applications, short video social applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0057] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with information input devices and information output, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 players (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4), portable computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, obtaining a set of original images), or it can be implemented as a single software or software module. No specific limitation is made here.

[0058] The server 105 may be a server that provides various services, such as a background server that processes the original image set obtained by the terminal devices 101, 102, and 103. The background server may perform corresponding processing based on the original image set obtained by the terminal device.

[0059] In some cases, the style image generation model training method provided in the present disclosure can be jointly performed by the terminal devices 101, 102, 103 and the server 105. For example, the step of "obtaining the original image set" can be performed by the terminal devices 101, 102, 103, and the step of "performing the training sample generation operation" can be performed by the server 105. The present disclosure does not limit this. Accordingly, the style image generation model training device can also be respectively set in the terminal devices 101, 102, 103 and the server 105.

[0060] In some cases, the style image generation model training method provided in the present disclosure may be executed by the server 105 , and accordingly, the style image generation model training device may also be arranged in the server 105 . In this case, the system architecture 100 may not include the terminal devices 101 , 102 , and 103 .

[0061] In some cases, the style image generation model training method provided in the present disclosure may be executed by the terminal devices 101 , 102 , and 103 . Accordingly, the style image generation model training apparatus may also be disposed in the terminal devices 101 , 102 , and 103 . In this case, the system architecture 100 may not include the server 105 .

[0062] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0063] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0064] In the related art, the method for training a style image generation model is generally to give a style-inducing word and bind the style and the style-inducing word so that the style image generation model learns the style. In the inference process of the style image generation model, as long as the text including the style-inducing word is input, a style image corresponding to the style can be generated.

[0065] However, when the style image generation model is trained using the above method, it is easy to learn the image content during the actual reasoning process of the style image generation model, making it difficult to decouple the image content and image style, resulting in the image output by the style image generation model being a non-style image.

[0066] Therefore, it is necessary to propose a new style image generation model training method to solve at least one of the above technical problems.

[0067] Continue to refer Figure 2A , Figure 2A A flowchart 200 of an embodiment of a style image generation model training method according to the present disclosure is shown. Figure 2A The style image generation model training method shown can be applied to Figure 1 The process 200 includes the following steps:

[0068] Step 201, obtaining a set of original images.

[0069] In this embodiment, each original image in the original image set conforms to the target style indicated by the target style keyword.

[0070] The target style may refer to an image style that the target style image generation model needs to output. The target style keyword may be a keyword indicating the target style.

[0071] For example, the target style keyword may be "Chinese style", "retro", "traditional", etc. Correspondingly, the target style indicated by the target style keyword may be Chinese classical style. Each original image in the original image set may be an image that conforms to Chinese classical style.

[0072] Step 202: For the original images in the original image set, a training sample generation operation is performed.

[0073] In this embodiment, the training sample generation operation may include: Figure 2B Steps 2021 to 2024 shown:

[0074] Step 2021, performing image content recognition on the original image to obtain styleless image content recognition text.

[0075] The image content may refer to specific elements in the image, for example, the image content may refer to elements such as mountains, water, trees, flowers, and birds in the image.

[0076] The text for identifying image content without style may refer to text related to image content without image style, for example, the text for identifying image content without style may refer to text containing mountains, water, trees, flowers, and birds in the image.

[0077] In some optional implementations, performing image content recognition on the original image to obtain the styleless image content recognition text may be inputting the original image into a pre-trained image content recognition model to obtain the styleless image content recognition text.

[0078] The image content recognition model can be used to recognize the image content of the original image and output styleless image content recognition text about the original image. The image content recognition model can be a pre-trained neural network model.

[0079] In some optional implementations, image content recognition is performed on each original image in the original image set to obtain the same styleless image content recognition text.

[0080] For example, if the image content of each original image includes mountains, water, trees, flowers, and birds, the same styleless image content recognition text will be output.

[0081] Step 2022: recognizing text based on the styleless image content and generating a styleless generated image set.

[0082] Among them, the styleless generated image may refer to an image that is only related to the image content but not to the image style.

[0083] For example, the generated non-styled images may all be images containing mountains, water, trees, flowers, and birds, and the image style of the images may not be Chinese classical style.

[0084] In some optional implementations, generating a set of styleless generated images based on the styleless image content recognition text may be inputting the styleless image content recognition text into a styleless image generation model to obtain the set of styleless generated images.

[0085] Here, the styleless image generation model is a model that has been trained with text and images and can generate styleless generated images by recognizing text through the content of styleless images, but cannot generate images with a specific style.

[0086] Here, the styleless image content recognition text may be input into the styleless image generation model at least once, that is, one styleless image content recognition text may generate multiple styleless generated images.

[0087] For example, a styleless image content recognition text can generate 10 styleless generated images. Each styleless generated image can be an image that meets the styleless image content recognition text, and the specific expressions of the image content of different styleless generated images can be different.

[0088] For example, the first unstyled generated image may be an image containing two mountains, a river, and two willow trees, and the second unstyled generated image may be an image containing one mountain, a river, and a peach tree.

[0089] In some optional implementations, the styleless image generation model may be a diffusion model (Diffusion Model).

[0090] Step 2023: determine the style image content text based on the target style keyword and the non-style image content recognition text.

[0091] In some optional implementations, determining the style image content text based on the target style keyword and the non-style image content recognition text may be concatenating the target style keyword and the non-style image content recognition text to obtain the style image content text.

[0092] In some optional implementations, concatenating the target style keywords and the non-style image content identification text may be directly concatenating the target style keywords and the non-style image content identification text.

[0093] For example, the keyword may be Chinese style, and the text of the non-styled image content recognition may be that the image contains mountains, water, trees, and birds. The style image content text obtained by directly splicing the target style keyword and the non-styled image content recognition text may be a Chinese style image with mountains, water, trees, and birds in the image.

[0094] In some optional implementations, concatenating the target style keywords and the non-style image content identification text to obtain the style image content text may also generate a natural language text containing the target style keywords and the non-style image content identification text.

[0095] For example, please generate an image with a Chinese style, in which the image contains tall mountains, lush trees, clear rivers, lush green trees, and birds flying high.

[0096] Step 2024, determining the training text and image supervision information in the styleless training sample based on the styleless image content recognition text and the styleless generated image, and determining the training text and image supervision information in the style training sample based on the style image content text and the original image.

[0097] In this embodiment, after obtaining the styleless image content recognition text, the styleless generated image, the style image content text and the original image through the aforementioned steps, the styleless image content text and the styleless generated image are paired to generate the training text and image supervision information in the styleless training sample, and the style image content text and the original image are paired to generate the training text and image supervision information in the style training sample.

[0098] Here, the training text of the style training sample can be image content recognition text, the image supervision information of the style training sample can be the original image, the training text of the non-style training sample can be non-style image content text, and the image supervision information of the non-style training sample can be a non-style generated image.

[0099] The training text and image supervision information in the styleless training sample can be in a 1:1 relationship. Correspondingly, the training text and image supervision information in the style training sample can also be in a 1:1 relationship. For example, by pairing a styleless image content recognition text with 10 styleless generated images, 10 pairs of styleless training samples can be obtained, and each pair of styleless training samples can include a training text and an image supervision information.

[0100] When there are 10 original images in the original image set, the 10 original images may be paired with the style image content texts respectively to generate 10 pairs of training texts and image supervision information in style training samples.

[0101] When there are 5 original images in the original image set, the original images can also be copied one by one to obtain 10 original images, and the 10 copied original images are paired with the style image content text to generate 10 pairs of training text and image supervision information in style training samples.

[0102] By configuring the training text and image supervision information in the non-style training samples and the training text and image supervision information in the style training samples to be in a 1:1 ratio, the training target style image generation model can be better treated for comparative training.

[0103] After step 202, at least one non-style training sample and at least one style training sample can be obtained.

[0104] Step 203: Based on at least one non-style training sample and at least one style training sample, a target style image generation model to be trained is trained to obtain a trained target style image generation model.

[0105] In this embodiment, after obtaining the training text and image supervision information of the non-style training samples and the training text and image supervision information in the style training samples, at least one non-style training sample and at least one style training sample are randomly selected from the non-style training samples and the style training samples, and based on the training text and image supervision information of the at least one non-style training sample and the training text and image supervision information of the at least one style training sample, the target style image generation model to be trained is trained to obtain the trained target style image generation model.

[0106] In some optional implementations, step 203 may include: Figure 2C Steps 2031A to 2032A shown:

[0107] Step 2031A: performing a model parameter adjustment operation on at least one non-style training sample and at least one style training sample.

[0108] The model parameter adjustment operation can be performed as follows: first, input the training text in at least one non-style training sample and at least one style training sample into the target style image generation model to be trained to obtain an actual generated image. Then, adjust the model parameters of the target style image generation model to be trained based on the difference between the actual generated image and the image supervision information in the at least one non-style training sample and at least one style training sample.

[0109] Inputting the training text in at least one non-style training sample and at least one style training sample into the target style image generation model to be trained may be randomly inputting the non-style training sample and the style training sample into the target style image generation model to be trained.

[0110] For example, a non-style training sample may be input for the first time to compare the difference between the generated actual image and the image supervision information of the non-style training sample, and a style training sample may be input for the second time to compare the difference between the generated actual image and the image supervision information in the style training sample. The model parameters of the target style image generation model to be trained are adjusted by comparing the difference between the generated actual image and the image supervision information of the non-style training sample, and by comparing the difference between the generated actual image and the image supervision information in the style training sample, and then the target style image generation model to be trained after the model parameters are adjusted is determined as the trained target style image generation model.

[0111] Step 2032A: determine the target style image generation model to be trained after the model parameters are adjusted as the trained target style image generation model.

[0112] Here, the trained target style image generation model is used to generate style images that match the target style.

[0113] In some optional implementations, step 203 may also include: Figure 2D Steps 2031B to 2035B are shown:

[0114] Step 2031B: performing a model parameter adjustment operation on at least one non-style training sample and at least one style training sample.

[0115] The model parameter adjustment operation can be performed as follows: first, input the training text in at least one non-style training sample and at least one style training sample into the target style image generation model to be trained to obtain an actual generated image. Then, adjust the model parameters of the target style image generation model to be trained based on the difference between the actual generated image and the image supervision information in the at least one non-style training sample and at least one style training sample.

[0116] Step 2032B: input the target style text in the target style text set into the target style image generation model after the model parameters are adjusted to obtain a target style generated image set.

[0117] The target style text set may be text containing target style keywords pre-stored in a text library. The target style text input into the target style image generation model after the model parameters are adjusted each time may be the same or different.

[0118] The target style generated images in the target style generated image set may be images that conform to the target style or images that do not conform to the target style.

[0119] Step 2033B, determining whether the proportion of images that conform to the target style in the target style generated image set is greater than or equal to a preset proportion threshold.

[0120] In some optional implementations, whether an image in the target-style generated image set conforms to the target style may be determined by manual evaluation.

[0121] For example, the images in the target style generated image set may be scored manually, and whether the images in the target style generated image set conform to the target style may be determined based on the scoring results.

[0122] In some optional embodiments, a method for determining whether an image in a target style generated image set conforms to the target style may also be to use a pre-trained model that can predict the degree of match between target style text and images, and use the model to determine whether an image in a target style generated image set conforms to the target style.

[0123] In addition, it is also possible to determine whether the images in the target style generated image set conform to the target style by referring to the matching degree between the image content in the target style generated image set and the styleless content recognition text of the image, and the image quality index, wherein the image quality index may be clarity, resolution, hue, etc.

[0124] After determining whether the images in the target style generated image set conform to the target style, further, the proportion of images conforming to the target style in the target style generated image set is obtained by calculation, and the proportion is used to determine whether the proportion of the number of images conforming to the target style is greater than or equal to a preset proportion threshold.

[0125] If the determination is yes, it indicates that the target style image generation model can already generate images that conform to the target style, and the training of the target style image generation model can be stopped, and the process can proceed to step 2034B for execution.

[0126] If the determination is no, it indicates that the proportion of the target style images generated by the target style image generation that conform to the target style is still low, and the target style image generation model needs to be further trained, so the process can proceed to step 2035B for execution.

[0127] Step 2034B, determining the target style image generation model after parameter adjustment as the trained target style image generation model.

[0128] Here, if it is determined in step 2033B that the target style image generation model can generate an image that conforms to the target style, the target style image generation model after parameter adjustment can be determined as the trained target style image generation model. Subsequently, the trained target style image generation model can be applied to generate the target style image.

[0129] Step 2035B, determining the target style image generation model after parameter adjustment as the trained target style image generation model.

[0130] Here, if it is determined in step 2033B that the target style image generation model needs to continue training, in order to increase the number of training samples, the images that conform to the target style in the target style generation image set can be added to the original image set, and the target style image generation model with adjusted parameters can be determined as the target style image generation model to be trained, and go to step 201 to continue executing the style image generation model training method.

[0131] The style image generation model training method provided by the embodiment of the present disclosure first obtains a set of original images, wherein the original images conform to the target style indicated by the target style keyword. Then, for the original images in the original image set, the following training sample generation operations are performed: image content recognition is performed on the original image to obtain a styleless image content recognition text, a styleless generated image set is generated based on the styleless image content recognition text, a style image content text is determined based on the target style keyword and the styleless image content recognition text, training text and image supervision information in the styleless training sample are determined based on the styleless image content recognition text and the styleless generated image, and training text and image supervision information in the styleless training sample are generated based on the style image content text and the original image. Finally, based on at least one styleless training sample and at least one style training sample, the target style image generation model to be trained is trained to obtain a trained target style image generation model. That is, the target style image generation model to be trained is trained by using the non-style training samples and the style training samples, and then the target style image generation model is obtained by comparative learning of the non-style training samples and the style training samples, so that when the target style image generation model learns the style of the original image, the image content and the image style can be better decoupled, thereby improving the effect of the target style image generation model in generating stylized images. In addition, the present disclosure can achieve the above-mentioned effect with a small number of training image samples, thereby reducing the training cost of the target style image generation model.

[0132] Further references Figure 3 , Figure 3 A flowchart of an embodiment of a method for generating a style image according to the present disclosure is shown. The process 300 includes the following steps:

[0133] Step 301: Obtain image content description text and style description text of the image to be generated.

[0134] Among them, the style description text is associated with the target style keywords.

[0135] In some optional implementations, the image content description text and the style description text may be embodied in one text.

[0136] For example, a landscape image with Chinese style and mountains, water, trees and birds in the picture.

[0137] In some optional implementations, the image content description text and the style description text may also be embodied by two texts.

[0138] For example, the image content description text and the style description text may be spliced ​​together to obtain a spliced ​​text including the image content description text and the style description text.

[0139] In some optional implementations, the image content description text and the style description text may also be obtained by parsing a piece of natural language text to obtain a style description text and an image content description text including target style keywords, and then splicing them together to obtain a spliced ​​text including the image content description text and the style description text.

[0140] Here, the image content description text may be provided by a user, and the style description text may be determined by a corresponding program, APP or software and provided to a target style image generation model.

[0141] For example, a style may be randomly selected through a program, APP or software, or multiple styles may be provided to the user. After the user determines one of the style options, the corresponding style description text is determined.

[0142] Step 302 : Based on the image content description text and the style description text, an image corresponding to the image content description text and the style description text is generated by a target style image generation model corresponding to the target style keyword.

[0143] The target style image generation model is used to generate images corresponding to the image content description text and style description text, where the target style image generation model is obtained through the above Figure 2A Or it can be obtained by training using the training method described in its optional implementation manner.

[0144] The image generated by the target style image generation model corresponding to the image content description text and the style description text can better decouple the image content and style. The target style image generation model can improve the effect of the target style image generation model in generating stylized images.

[0145] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a style image generation model training device, and the device embodiment is Figure 2ACorresponding to the method embodiment shown, the device can be specifically applied to various terminal devices and servers.

[0146] like Figure 4 As shown, the style image generation model training device of this embodiment, the device 400 includes: an original image set acquisition unit 401, a training sample generation unit 402 and a target style image generation model training unit 403. Among them, the original image set acquisition unit 401 is used to acquire an original image set, wherein the original image meets the target style indicated by the target style keyword; the training sample generation unit 402 is used to perform the following training sample generation operations on the original images in the original image set: perform image content recognition on the original image to obtain a styleless image content recognition text; generate a styleless generated image set based on the styleless image content recognition text; determine the style image content text based on the target style keyword and the styleless image content recognition text; determine the training text and image supervision information in the styleless training sample based on the styleless image content recognition text and the styleless generated image, and determine the training text and image supervision information in the style training sample based on the style image content text and the original image; and the target style image generation model training unit 403 is used to train the target style image generation model to be trained based on at least one of the styleless training samples and at least one of the style training samples to obtain a trained target style image generation model.

[0147] In this embodiment, the specific processing of the original image set acquisition unit 401, the training sample generation unit 402 and the target style image generation model training unit 403 of the style image generation model training device and the technical effects thereof can be referred to respectively. Figure 2A The relevant descriptions of step 201 to step 203 in the corresponding embodiment are not repeated here.

[0148] In some optional implementations, the training sample generating unit 402 may be further configured as follows:

[0149] The target style keyword and the non-style image content recognition text are concatenated to obtain the style image content text.

[0150] In some optional implementations, the training sample generating unit 402 may be further configured as follows:

[0151] The styleless image content recognition text is input into the styleless image generation model to obtain the styleless generated image set.

[0152] In some optional implementations, the target style image generation model training unit 403 may be further configured as follows:

[0153] For at least one of the non-style training samples and at least one of the style training samples, the following model parameter adjustment operation is performed: inputting the training text in at least one of the non-style training samples and at least one of the style training samples into the target style image generation model to be trained to obtain an actual generated image; adjusting the model parameters of the target style image generation model to be trained based on the difference between the actual generated image and the image supervision information in at least one of the non-style training samples and at least one of the style training samples;

[0154] The target style image generation model to be trained after the model parameters are adjusted is determined as the trained target style image generation model.

[0155] In some optional implementations, the target style image generation model training unit 403 may be further configured as follows:

[0156] For at least one of the non-style training samples and at least one of the style training samples, the following model parameter adjustment operation is performed: inputting the training text in at least one of the non-style training samples and at least one of the style training samples into the target style image generation model to be trained to obtain an actual generated image; adjusting the model parameters of the target style image generation model to be trained based on the difference between the actual generated image and the image supervision information in at least one of the non-style training samples and at least one of the style training samples;

[0157] The target style text in the target style text set is input into the target style image generation model after the model parameters are adjusted to obtain a trained target style image generation model.

[0158] In some optional implementations, the target style image generation model training unit 403 may be further configured as follows:

[0159] Inputting the target style text in the target style text set into the target style image generation model after the model parameters are adjusted to obtain a target style generated image set;

[0160] Determine whether the proportion of images that meet the target style in the target style generated image set is greater than or equal to a preset proportion threshold;

[0161] In response to the determination being yes, determining the target style image generation model after the parameter adjustment as the trained target style image generation model;

[0162] In response to a no determination, the images in the target style generated image set that conform to the target style are added to the original image set, and the target style image generation model after the parameter adjustment is determined as the target style image generation model to be trained, and the style image generation model training method is continued.

[0163] In some optional implementations, the training sample generating unit 402 may be further configured as follows:

[0164] The original image is input into a pre-trained image content recognition model to obtain styleless image content recognition text.

[0165] It should be noted that the implementation details and technical effects of each unit in the style image generation model training device provided in the embodiments of the present disclosure can be referred to the description of other embodiments in the present disclosure, and will not be repeated here.

[0166] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a style image generation device, and the device embodiment is Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various terminal devices and servers.

[0167] like Figure 5 As shown, the style image generation device of this embodiment, the device 500 includes: an image content description text and style description text acquisition unit 501 and an image generation unit 502. The image content description text and style description text acquisition unit 501 is used to acquire the image content description text and style description text of the image to be generated, wherein the style description text is associated with a target style keyword; the image generation unit 502 is used to generate an image corresponding to the image content description text and style description text based on the image content description text and style description text, through a target style image generation model corresponding to the target style keyword, wherein the target style image generation model is adopted as shown in the figure. Figure 2A Or it is obtained by pre-training using the method described in the optional implementation manner.

[0168] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing the present disclosure. Figure 6 The computer system 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0169] like Figure 6As shown, the computer system 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. Various programs and data required for the operation of the computer system 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 604 is also connected to the bus 604.

[0170] Typically, the following devices may be connected to the I / O interface 604: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication devices 609 may allow the computer system 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The computer system 600 of an electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0171] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0172] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0173] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0174] The computer readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements the following Figure 2A and / or Figure 3 The illustrated embodiment and its optional implementations illustrate a style image generation model training method and / or a style image generation method.

[0175] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0176] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0177] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the original image set acquisition unit may also be described as a "unit for acquiring an original image set".

[0178] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. A style image generation model training method, characterized in that: The method comprises: Acquire an original image set, wherein the original images conform to a target style indicated by a target style keyword; For the original image in the original image set, the following training sample generation operations are performed: image content recognition is performed on the original image to obtain a styleless image content recognition text; based on the styleless image content recognition text, a styleless generated image set is generated; based on the target style keyword and the styleless image content recognition text, a style image content text is determined; based on the styleless image content recognition text and the styleless generated image, training text and image supervision information in the styleless training sample are determined, and based on the style image content text and the original image, training text and image supervision information in the style training sample are determined; Based on at least one of the non-style training samples and at least one of the style training samples, a target style image generation model to be trained is trained to obtain a trained target style image generation model.

2. The method according to claim 1, characterized in that The determining of the style image content text based on the target style keyword and the non-style image content recognition text includes: The target style keyword and the non-style image content recognition text are concatenated to obtain the style image content text.

3. The method according to claim 1, characterized in that The step of recognizing text based on the content of the styleless image and generating a set of styleless generated images includes: The styleless image content recognition text is input into the styleless image generation model to obtain the styleless generated image set.

4. The method according to claim 1, characterized in that: The step of training a target style image generation model to be trained based on at least one of the non-style training samples and at least one of the style training samples to obtain a trained target style image generation model includes: For at least one of the non-style training samples and at least one of the style training samples, the following model parameter adjustment operation is performed: inputting the training text in at least one of the non-style training samples and at least one of the style training samples into the target style image generation model to be trained to obtain an actual generated image; adjusting the model parameters of the target style image generation model to be trained based on the difference between the actual generated image and the image supervision information in at least one of the non-style training samples and at least one of the style training samples; The target style image generation model to be trained after the model parameters are adjusted is determined as the trained target style image generation model.

5. The method according to claim 1, characterized in that The step of training a target style image generation model to be trained based on at least one of the non-style training samples and at least one of the style training samples to obtain a trained target style image generation model includes: For at least one of the non-style training samples and at least one of the style training samples, the following model parameter adjustment operation is performed: inputting the training text in at least one of the non-style training samples and at least one of the style training samples into the target style image generation model to be trained to obtain an actual generated image; adjusting the model parameters of the target style image generation model to be trained based on the difference between the actual generated image and the image supervision information in at least one of the non-style training samples and at least one of the style training samples; The target style text in the target style text set is input into the target style image generation model after the model parameters are adjusted to obtain a trained target style image generation model.

6. The method according to claim 5, characterized in that The step of inputting the target style text in the target style text set into the target style image generation model after adjusting the model parameters to obtain the trained target style image generation model comprises: Inputting the target style text in the target style text set into the target style image generation model after the model parameters are adjusted to obtain a target style generated image set; Determine whether the proportion of images that meet the target style in the target style generated image set is greater than or equal to a preset proportion threshold; In response to the determination being yes, determining the target style image generation model after the parameter adjustment as the trained target style image generation model; In response to a no determination, the images in the target style generated image set that conform to the target style are added to the original image set, and the target style image generation model after the parameter adjustment is determined as the target style image generation model to be trained, and the style image generation model training method is continued.

7. The method according to claim 1, characterized in that The performing image content recognition on the original image to obtain styleless image content recognition text includes: The original image is input into a pre-trained image content recognition model to obtain styleless image content recognition text.

8. A style image generation method, characterized in that: The method comprises: Obtaining image content description text and style description text of the image to be generated, wherein the style description text is associated with a target style keyword; Based on the image content description text and the style description text, an image corresponding to the image content description text and the style description text is generated through a target style image generation model corresponding to the target style keyword, wherein the target style image generation model is pre-trained using the method described in any one of claims 1 to 7.

9. A style image generation model training device, comprising: An original image set acquisition unit, used to acquire an original image set, wherein the original images conform to the target style indicated by the target style keyword; A training sample generating unit is used to perform the following training sample generating operations on an original image in the original image set: performing image content recognition on the original image to obtain a styleless image content recognition text; generating a styleless generated image set based on the styleless image content recognition text; determining a style image content text based on the target style keyword and the styleless image content recognition text; determining training text and image supervision information in a styleless training sample based on the styleless image content recognition text and the styleless generated image, and determining training text and image supervision information in a style training sample based on the style image content text and the original image; The target style image generation model training unit is used to train the target style image generation model to be trained based on at least one of the non-style training samples and at least one of the style training samples to obtain a trained target style image generation model.

10. A style image generating device, comprising: An image content description text and style description text acquisition unit, used to acquire an image content description text and a style description text of a to-be-generated image, wherein the style description text is associated with a target style keyword; An image generation unit is used to generate an image corresponding to the image content description text and the style description text based on the image content description text and the style description text through a target style image generation model corresponding to the target style keyword, wherein the target style image generation model is pre-trained using the method described in any one of claims 1 to 7.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7 and / or 8.

12. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 7 and / or 8 is implemented.

Images