Model training method, image generation method and device
By using original image and text description information in the image generation model to generate the target image and reconstruct the prior image through label information, the problems of image diversity and fidelity in the prior art are solved, and high-fidelity diversified images are achieved. generate.
Patent Information
- Application Number
- CN202410917610.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-08
AI Technical Summary
When the current literary and biographical image technology generates diverse images, the diversity effect is poor, and it cannot fully reflect the real performance of the target object in different scenarios or forms, and may lead to alienation distortion of the target object, affecting the authenticity and consistency of the image. sex.
By obtaining the original image and text description information, input it into the image generation model to generate the target image, and reconstruct the prior image using the label information, determine the first deviation and the second deviation to minimize these deviations to perform the image generation model Training to generate both fidelity and diverse target images.
It improves the image expansion ability of the image generation model during multi-dimensional creation, fully protects the feature information of the original designated object, and significantly improves the fidelity and consistency of the target image.
Smart Images

Figure CN118918201A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of image processing, and in particular to a model training method, an image generation method and a device. Background Art
[0002] At present, with the rapid development and progress in the field of artificial intelligence, text image technology has become more and more mature. It can efficiently create rich and diverse high-quality images by understanding and transforming text data. Text image technology is becoming increasingly popular and widely used in many industries, such as creative design, media publicity, personal information change, etc. The practicality and convenience it brings in various fields are unparalleled.
[0003] In the current Wenshengtu technology, in order to create diversified images for a certain target object, the example pictures with the target object are mainly used as the reference basis, and then the corresponding text description is applied to generate diversified images based on the target object. However, when the Wenshengtu technology at this stage generates diversified images for the target object, the diversity effect of the generated images is poor, and it is not possible to well show the actual performance of the target object in different scenes or forms. In addition, the images generated by this means may also be alienated and distorted to the target object in the original example image, which will have a negative impact on the authenticity of the target object and the consistency with the original example image.
[0004] Therefore, how to generate high-fidelity and diverse rich multi-images for the target object is an urgent problem to be solved. Summary of the invention
[0005] This specification provides a model training method, an image generation method and a device to partially solve the above-mentioned problems existing in the prior art.
[0006] This manual adopts the following technical solutions:
[0007] This manual provides a model training method, including:
[0008] Acquire an original image containing a specified object and text description information, wherein the text description information is used to represent the image content of a target image containing the specified object generated based on the original image;
[0009] Inputting the original image and the text description information into an image generation model to be trained, so that the image generation model generates the target image based on the original image;
[0010] Inputting the text description information into the image generation model so that the image generation model generates a priori image corresponding to the text description information, and determining label information corresponding to the original image, wherein the label information is used to represent the specified object contained in the original image, and inputting the label information and the priori image into the image generation model so that the image generation model reconstructs the priori image according to the label information to obtain a reconstructed priori image;
[0011] Determining reference data for the target image, and determining a first deviation based on the reference data and the target image, and determining a second deviation based on the prior image and the reconstructed prior image;
[0012] The image generation model is trained with minimizing the first deviation and the second deviation as an optimization goal.
[0013] Optionally, inputting the original image and the text description information into an image generation model to be trained so that the image generation model generates a target image based on the original image specifically includes:
[0014] The original image and the text description information are input into the image generation model, so that the image generation model determines the original feature data corresponding to the original image based on the original image, and determines the text feature data corresponding to the text description information based on the text description information, and determines the fused feature data for the original image and the text description information based on the original feature data and the text feature data, and generates the target image based on the fused feature data.
[0015] Optionally, determining reference data for the target image specifically includes:
[0016] Determining original feature data corresponding to the original image according to the original image;
[0017] Determining text feature data corresponding to the text description information according to the text description information;
[0018] Determining reference data for the target image according to the original feature data and the text feature number;
[0019] Determining a first deviation according to the reference data and the target image specifically includes:
[0020] According to the target image, determining target feature data corresponding to the target image;
[0021] A deviation between the target feature data and the reference data is determined as the first deviation.
[0022] Optionally, determining a second deviation according to the priori image and the reconstructed priori image specifically includes:
[0023] Determining, according to the reconstructed priori image, reconstructed feature data corresponding to the reconstructed priori image;
[0024] Determine, based on the prior image, prior feature data corresponding to the prior image, and determine, based on the label information, label feature data corresponding to the label information;
[0025] Determining reconstruction reference data for the reconstructed prior image according to the prior feature data and the label feature data;
[0026] The second deviation is determined according to the reconstructed feature data and the reconstructed reference data.
[0027] Optionally, determining the second deviation according to the reconstructed feature data and the reconstructed reference data specifically includes:
[0028] determining a deviation between the reconstructed feature data and the reconstructed reference data as an intermediate deviation;
[0029] Inputting the prior image and the original image into a preset image similarity calculation model so that the image similarity calculation model determines a similarity value between the prior image and the original image;
[0030] The second deviation is determined according to the intermediate deviation and the similarity value.
[0031] This specification provides a method for generating an image, comprising:
[0032] receiving an image creation request;
[0033] According to the image creation request, an original image including a specified object and text description information required for creating an image are acquired, wherein the text description information is used to represent the image content of a target image including the specified object generated based on the original image;
[0034] The original image and the text description information are input into an image generation model so that the image generation model generates the target image based on the original image. The image generation model is trained by the above-mentioned model training method.
[0035] This specification provides a model training device, including:
[0036] An acquisition module, used for acquiring an original image containing a specified object and text description information, wherein the text description information is used for representing the image content of a target image containing the specified object generated based on the original image;
[0037] A generation module, used for inputting the original image and the text description information into an image generation model to be trained, so that the image generation model generates the target image based on the original image;
[0038] A reconstruction module, used for inputting the text description information into the image generation model so that the image generation model generates a priori image corresponding to the text description information, and determining label information corresponding to the original image, wherein the label information is used to represent the specified object contained in the original image, and inputting the label information and the priori image into the image generation model so that the image generation model reconstructs the priori image according to the label information to obtain a reconstructed priori image;
[0039] a deviation determination module, configured to determine reference data for the target image, and determine a first deviation based on the reference data and the target image, and determine a second deviation based on the prior image and the reconstructed prior image;
[0040] A training module is used to train the image generation model with minimizing the first deviation and the second deviation as an optimization goal.
[0041] Optionally, the generation module is specifically used to input the original image and the text description information into the image generation model, so that the image generation model determines the original feature data corresponding to the original image based on the original image, and determines the text feature data corresponding to the text description information based on the text description information, and determines the fused feature data for the original image and the text description information based on the original feature data and the text feature data, and generates the target image based on the fused feature data.
[0042] Optionally, the deviation determination module is specifically used to determine original feature data corresponding to the original image according to the original image; determine text feature data corresponding to the text description information according to the text description information; determine reference data for the target image according to the original feature data and the text feature number;
[0043] The deviation determination module is specifically used to determine, based on the target image, target feature data corresponding to the target image; and determine a deviation between the target feature data and the reference data as the first deviation.
[0044] Optionally, the deviation determination module is specifically used to determine, based on the reconstructed prior image, reconstructed feature data corresponding to the reconstructed prior image; determine, based on the prior image, prior feature data corresponding to the prior image, and, based on the label information, determine label feature data corresponding to the label information; determine, based on the prior feature data and the label feature data, reconstructed reference data for the reconstructed prior image; and determine the second deviation based on the reconstructed feature data and the reconstructed reference data.
[0045] Optionally, the deviation determination module is specifically used to determine the deviation between the reconstructed feature data and the reconstructed reference data as an intermediate deviation; input the prior image and the original image into a preset image similarity calculation model so that the image similarity calculation model determines the similarity value between the prior image and the original image; and determine the second deviation based on the intermediate deviation and the similarity value.
[0046] This specification provides an image generation device, including:
[0047] A receiving module, used for receiving an image creation request;
[0048] an acquisition module, configured to acquire, according to the image creation request, an original image including a specified object and text description information required for creating an image, wherein the text description information is used to represent the image content of a target image including the specified object generated based on the original image;
[0049] An image generation module is used to input the original image and the text description information into an image generation model so that the image generation model generates the target image based on the original image. The image generation model is trained by the model training method as described above.
[0050] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned model training method and image generation method.
[0051] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned model training method and image generation method when executing the program.
[0052] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:
[0053] It can be seen from the above method that in the model training method and the image generation method provided in this specification, the acquired original image containing the specified object and the text description information can be input into the image generation model to be trained, so that the image generation model can generate the target image containing the specified object. Next, the text description information can be input into the image generation model alone, so that the image generation model can generate the corresponding prior image according to the text description information, and at the same time determine the label information corresponding to the original image, and then input the label information and the prior image into the image generation model, so that the image generation model can generate a reconstructed prior image of the prior image according to the label information. Finally, the first deviation is determined according to the target image and the reference data of the target image, and the second deviation is determined according to the prior image and the reconstructed prior image, and the image generation model is trained with minimizing the first deviation and the second deviation as the optimization goal. The trained image generation model is used to generate images according to the original image and the text description information.
[0054] From the above content, it can be seen that the model training method and image generation method provided in this specification can generate corresponding prior images according to text description information, and reconstruct the prior images through the image generation model according to the label information of the original image, and then train and optimize the image generation model according to the reconstructed prior images and the prior images. The image generation model trained by this method can generate diversified images that not only retain the specified objects in the original image, but also highly conform to the text description information, effectively improving the image expansion ability of the image generation model when performing multivariate image creation for the specified object, while completely protecting the integrity of the feature information of the original specified object, and greatly improving the fidelity of the specified object in the generated target image. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:
[0056] Figure 1 A flowchart of a model training method provided in this specification;
[0057] Figure 2 This is a schematic diagram of an example of an original image provided in this specification;
[0058] Figure 3 This is a schematic diagram of an example of a target image provided in this specification;
[0059] Figure 4 This is a schematic diagram of an example of a priori image provided in this specification;
[0060] Figure 5 A schematic diagram of a flow chart of an image generation method provided in this specification;
[0061] Figure 6 A schematic diagram of a model training device provided in this specification;
[0062] Figure 7 A schematic diagram of an image generating device provided in this specification;
[0063] Figure 8 A method provided in this specification corresponding to Figure 1 and Figure 5 Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION
[0064] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.
[0065] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.
[0066] Figure 1 A flow chart of a model training method provided in this specification includes the following steps:
[0067] S101: Acquire an original image containing a specified object and text description information.
[0068] With the continuous advancement of science and technology, text-based image technology has become more and more perfect. The ability to generate various high-quality images based on text understanding has demonstrated its extremely wide application value in various industries. However, in the process of realizing diversified image creation for specific objects, the current text-based image technology still has some obvious defects. The diversity effect of images generated by current technology is poor, and it is difficult to fully reflect the true performance of specified objects under various environmental conditions and morphological changes. In addition, in the process of generating the final image, there may be special circumstances where the specified object undergoes a certain degree of deformation or distortion, thereby affecting the authenticity of the specified object in the generated image. Therefore, it is crucial to generate diverse, real and rich images for specified objects.
[0069] To this end, this specification provides a model training method and an image generation method, wherein the execution subject of the method provided in this specification can be a terminal device such as a desktop computer, a laptop computer, or a server. In addition, the execution subject of this specification can also be a subject in the form of software, such as a client installed in a terminal device. For the sake of convenience, this specification below only uses the terminal device as the execution subject to describe the provided model training method and image generation method.
[0070] Based on this, the terminal device that applies the model training method and the image generation method provided in this specification can generate a target image that conforms to the description content in the text description information based on the original image and text description information obtained. Among them, the application scenario when the terminal device performs diversified image creation based on the original image and text description information can be determined according to actual needs. For example, in the scenario of introducing and promoting the goods to be sold, the merchant of the goods often needs to carry out diversified promotion of the goods to be sold. The terminal device can generate diversified images that meet the merchant's expectations and conform to the actual characteristics of the goods based on the example images of the goods input by the merchant and the text description information corresponding to the performance style or form of the goods to be sold when promoting the goods, so that the merchant can promote and introduce the goods to be sold based on the obtained diversified images, which effectively improves the merchant's promotion efficiency for the goods to be sold.
[0071] For another example, in the scenario of identity authentication based on face images, the terminal device can obtain the user's face sample image in advance, and generate the user's face image in different scenes or forms according to the corresponding text description. Then, when the identity authentication is actually performed based on the real face image, the user's identity information in different scenes or states can be judged based on the pre-generated diversified face images, which effectively improves the verification efficiency and accuracy of the identity authentication link.
[0072] The method provided in this specification is mainly divided into two stages, one is the model training stage, and the other is the actual application stage. In the model training stage, the terminal device can obtain the original image containing the specified object, as well as the text description information used to represent the image content of the target image containing the specified object generated based on the original image in the subsequent process.
[0073] In order to facilitate understanding of the specific content and form of the above original image and text description information, the following is an example of an original image and the corresponding text description. Figure 2 shown.
[0074] Figure 2 This is a schematic diagram of an example of an original image provided in this specification.
[0075] like Figure 2 As shown in , the terminal device can obtain Figure 2 The original image in the figure is a kitten drinking water. The kitten contained in the original image is the specified object. The corresponding text description information can be determined according to the actual generation requirements. Figure 2 In the original image, it is necessary to generate an image of a designated object "kitten" sleeping, and the corresponding text description information can be text information such as "a kitten is sleeping". It should be noted that the specific content of the original image and the text description information is not limited in this specification. The text description information only needs to keep the designated object in the original image as the main body, and the specific text content and modified content can be flexibly adapted according to actual needs. For example, the text description information of "a kitten is sleeping" in the above example can also be similar to "a kitten is sleeping on the grass", "a kitten is sleeping on its back", etc.
[0076] S102: Inputting the original image and the text description information into an image generation model to be trained, so that the image generation model generates the target image based on the original image.
[0077] In this specification, the terminal device can input the acquired original image and text description information into the image generation model to be trained, so that the image generation model can generate a corresponding target image based on the original image according to the image content described by the text description information.
[0078] Specifically, after obtaining the original image and text description information, the image generation model to be trained can determine the original feature data corresponding to the original image based on the original image, and the original feature data is used to represent the image features in the original image. Then, the image generation model can also determine the text feature data corresponding to the text description information based on the text description information, and the text feature data is used to represent the feature information corresponding to the text description information.
[0079] Next, the image generation model can determine the fused feature data after fusing the original image with the feature data in the text description information based on the original feature data corresponding to the original image and the text feature data corresponding to the text description information, and generate a target image based on the fused feature data that can both satisfy the image description in the text description information and retain the image features of the specified object in the original image.
[0080] In order to facilitate the understanding of the generation process of the above target image, in the above example Figure 2 Based on this, we will introduce an example diagram of a target image. Figure 3 shown.
[0081] Figure 3 This is a schematic diagram of an example of a target image provided in this specification.
[0082] like Figure 3 As shown in , it is assumed that the terminal device obtains Figure 2 The original image and the text description of "a kitten sleeping" in the terminal device can Figure 2 The original image shown in the figure and “a kitten sleeping” are input into the image generation model to be trained, so that the image generation model can generate the following Figure 3 The target image shown in not only satisfies the image description in the text description information (a kitten is sleeping), but also retains the image features of the specified object (kitten) in the original image.
[0083] It should be noted that the specific type and structure of the image generation model are not strictly limited in this specification. The process of determining the original feature data based on the original image can use a mathematical model with data conversion capability such as an autoencoder (AE). The process of determining the text feature data based on the text description information can use a mathematical model such as the BERT model (Bidirectional Encoder Representations from Transformers) that converts text data into vector data. The subsequent generation process of generating the target image based on the text feature data and the original feature data can use a mathematical model with image generation capability such as the U-NET model. The models or tools used in all the above-mentioned processes are not strictly limited in this specification and can be flexibly selected and set according to actual application scenarios and requirements.
[0084] S103: Input the text description information into the image generation model so that the image generation model generates a prior image corresponding to the text description information, and determines the label information corresponding to the original image, wherein the label information is used to represent the specified object contained in the original image, and input the label information and the prior image into the image generation model so that the image generation model reconstructs the prior image according to the label information to obtain a reconstructed prior image.
[0085] In this specification, the terminal device can input the text description information into the image generation model to be trained separately, so that the image generation model can generate the corresponding prior image according to the image content described by the text description information. The generated prior image is used to improve the diversified ability of the image generation model when generating the target image according to the original image and the text description information in the subsequent model training optimization process.
[0086] While generating the prior image corresponding to the text description information, the terminal device can determine the label information corresponding to the specified object in the original image according to the image content of the original image. Then, the terminal device can input the label information and the prior image generated by the above process into the image generation model, so that the image generation model reconstructs the prior image according to the label information and generates a reconstructed prior image corresponding to the prior image.
[0087] Specifically, the specific process of reconstructing the prior image corresponding to the prior image according to the label information by the above-mentioned image generation model is similar to the process of generating the target image in the above-mentioned steps. The image generation model can determine the prior feature data corresponding to the prior image according to the prior image, and the prior feature data is used to represent the image features in the prior image, and determine the label feature data corresponding to the label information according to the label information, and the label feature data is used to represent the feature information corresponding to the label information. The label information can be determined by the terminal device according to the image content of the original image itself, or it can be generated by actively marking the original image manually. The method of generating the label information is not strictly limited in this specification.
[0088] Then, the image generation model can fuse the prior feature data and the label feature data, and reconstruct the reconstructed prior image corresponding to the prior image based on the fused data. Figure 2 Based on this, the following is an example image of a priori image for introduction and explanation. For details, please refer to Figure 4 as shown in .
[0089] Figure 4 This is a schematic diagram of an example of a priori image provided in this specification.
[0090] Figure 4 The prior image shown in is the terminal device sending the instance Figure 2 The corresponding text description information "a kitten is sleeping" is generated after being input into the image generation model. Figure 4 It is not difficult to see that Figure 4 The prior images in not only contain the morphological image of the kitten when it is sleeping, but also contain some other diversified image features, such as Figure 4 The terminal device can determine the label information corresponding to the original image while generating the prior image corresponding to the text description information. Figure 2 For example, the terminal device can Figure 2 The original image in the image is determined to have a label of "a kitten". Then, the terminal device can Figure 4The prior image and Figure 2 The label information "a kitten" corresponding to the original image is input into the image generation model, so that the image generation model can reconstruct the prior image according to the label information corresponding to the original image, and obtain the reconstructed prior image corresponding to the prior image. The purpose of reconstructing the prior image here is specifically to enable the image processing model to learn and record the diversified feature data in the prior image when the image processing model is subsequently trained and optimized according to the prior image and the reconstructed prior image, and to improve the ability to effectively combine the learned diversified feature data in the prior image with the specified object in the original image.
[0091] It should be noted that in the process of the terminal device generating a priori images according to the text description information mentioned above, the terminal device can generate multiple prior images, so as to achieve better training effect in the subsequent training process of the image generation model. In this specification, there is no strict limit on the number of prior images generated, and it can be flexibly set according to actual needs. In addition, when generating multiple prior images, the terminal device can reconstruct each prior image through the image generation model according to the label information of the original image to obtain a reconstructed prior image corresponding to each prior image.
[0092] S104: Determine reference data for the target image, and determine a first deviation based on the reference data and the target image, and determine a second deviation based on the prior image and the reconstructed prior image.
[0093] S105: Training the image generation model with minimizing the first deviation and the second deviation as an optimization goal.
[0094] In this specification, the terminal device can determine the reference data for the target image based on the original feature data corresponding to the original image and the text feature data corresponding to the text description information, and determine the target feature data corresponding to the target image based on the target image. The reference data for the target image is used to represent the feature data corresponding to the target image generated based on the original image according to the text description information under an ideal state. Then, the terminal device can determine the deviation between the reference data and the target feature data based on the reference data and the target feature data, and use it as the first deviation. For details, refer to the following formula:
[0095]
[0096] in, Used to indicate the first deviation, z I Used to represent the original feature data corresponding to the original image, T IUsed to represent the text feature data corresponding to the text description information, {(z I ,T I )} represents the reference data for the target image, and z is used to represent the target feature data corresponding to the target image. ∈ is used to represent the real noise sampling data when the image generation model generates the image, ∈ θ (z t ,t,Γ(T I )) is used to represent the noise sampling data at each moment in the image generation process, z t It is represented as the noise image feature data representing each moment in the image generation process. t is used to represent the moment information, which is obtained by sampling from the uniform distribution [0,1]. Γ(T I ) is used to represent the data representation of text feature data in the image generation process.
[0097] The specific expression of the overall formula is: by calculating the real noise sampling data ∈ and the noise data ∈ when the image generation model generates the image θ The Euclidean norm between And according to the reference data {(z I ,T I )} and target feature data z are averaged over all time information t to obtain the deviation between the reference data and the target feature data As the first deviation.
[0098] Furthermore, the terminal device can determine the reconstructed feature data corresponding to the reconstructed prior image based on the reconstructed prior image corresponding to the prior image, and the label feature data corresponding to the prior feature data and the label information corresponding to the prior image. Determine the reconstructed reference data corresponding to the reconstructed prior image. The terminal device can determine the deviation between the reconstructed feature data and the reconstructed reference data based on the reconstructed reference data and the reconstructed feature data as an intermediate deviation. Then, the terminal device can input the prior image and the original image together into a preset image similarity calculation model, so that the image similarity calculation model can determine the similarity value between the prior image and the original image. Then, based on the similarity value and the intermediate deviation, determine the second deviation. For details, please refer to the following formula:
[0099]
[0100] in, Used to indicate the second deviation, It is used to represent the intermediate deviation between the reconstructed feature data and the reconstructed reference data, z CS It is used to represent the prior feature data corresponding to the prior image, T CS Used to represent the label feature data corresponding to the label information, {(z CS,T CS )} represents the reconstruction reference data for reconstructing the prior image, z CS′ It is used to represent the reconstructed feature data corresponding to the reconstructed prior image. ∈ is used to represent the real noise sampling data when the image generation model reconstructs the prior image. θ (z,t,Γ(T CS )) is used to represent the noise sampling data at each moment in the image reconstruction process, z is represented as the noise image feature data at each moment in the image generation process, t is used to represent the time information, which is obtained by sampling from the uniform distribution [0,1], Γ(T CS ) is used to represent the data representation of label feature data in the image generation process.
[0101] The specific expression of the overall formula in the first line above is that by calculating the real noise sampling data ∈ and the noise data ∈ when the image is reconstructed by the image generation model θ The Euclidean norm between And according to the reconstructed reference data {(z CS ,T CS )} and reconstruct feature data z CS′ All the time information t is averaged to obtain the deviation between the reconstructed reference data and the reconstructed feature data. Then, the terminal device can determine the similarity value λ between the prior image and the original image through the preset image similarity calculation model k , similarity value λ k The value range is (0,1), and the similarity value λ k Deviation from the middle The product of is taken as the second deviation.
[0102] It should be noted that K in the above formula is used to represent the number of prior images. When multiple prior images are generated through the above steps, the terminal device can determine the second deviation between each pair of prior images and the reconstructed prior images based on the label information of each prior image and the original image, and the corresponding reconstructed prior images, and determine the sum of the second deviations corresponding to all prior images as the total second deviation.
[0103] It should also be noted that the similarity value λ in the above formula k It can represent the similarity between the original image and the prior image. The diversified feature data in the prior image with a higher similarity than the original image can be better integrated with the original image. The similarity value can be understood as a weight coefficient for each intermediate deviation. The optimization process of the second deviation corresponding to the prior image with a larger similarity value can more effectively improve the diversified ability of the image generation model in generating target images in the subsequent training process.
[0104] In addition, in this specification, the specific type of image similarity calculation model is not limited. Mathematical models with image feature analysis capabilities such as the DINO model (Deep Visual Self-Supervised Learning with Noisy-Student) can be used, and can be flexibly set according to actual application scenarios and needs.
[0105] Furthermore, the terminal device may train the image generation model with minimizing the first deviation and the second deviation as the optimization goal, and specifically refer to the following formula:
[0106]
[0107] in, Used to indicate the first deviation, Used to indicate the second deviation, when the terminal device generates multiple prior images, It is used to represent the sum of the second deviations corresponding to all prior images, that is, the total second deviation mentioned above. The terminal device can determine the first deviation With the second deviation The sum value deviation L is then used as the optimization goal to train the image generation model to obtain the trained image generation model for performing the image generation task.
[0108] The method provided in this specification is mainly divided into two stages, the model training stage and the actual application stage. The model training stage introduced above is mainly used to obtain an image generation model with diversified image generation functions after model training, so that in the actual application stage, the image generation model can be applied to a server or terminal device for performing image generation tasks. In order to facilitate the introduction of the image generation method, the following will be explained with a flow chart of the image generation method, as shown in FIG. Figure 5 shown.
[0109] Figure 5 A schematic diagram of a method for generating an image provided in this specification includes the following steps:
[0110] S501: Receive an image creation request.
[0111] S502: According to the image creation request, an original image including a specified object and text description information required for creating the image are obtained.
[0112] At present, with the rapid progress of various scientific and technological fields, the text image technology has gradually matured and is widely used in various fields. However, when the current text image technology is used to create a variety of target objects based on the instance images and text descriptions containing the target objects, the generated images cannot well reflect the diversified effects of the target objects. In addition, there may be a situation where the generated image modifies and mutates the target objects in the original instance images, resulting in the fidelity of the target objects in the generated images being affected. Therefore, how to generate diverse and real rich images for specified objects is an extremely important issue.
[0113] To this end, this specification provides a method for generating an image, wherein the execution subject adopted by the method provided in this specification can be a server, or a terminal device such as a desktop computer, a laptop computer, etc., and in addition, the execution subject of this specification can also be a subject in the form of software, such as a client installed in a terminal device, etc. For the sake of convenience of description, this specification below only takes the terminal device as the execution subject to describe the provided method for generating an image.
[0114] Based on this, the terminal device using the image generation method provided in this specification can generate a target image that satisfies the image content described in the text description information and fully embodies the image features of the specified object in the original image based on the original image and the text description information. Among them, the application scenario when the terminal device performs diversified image creation based on the original image and the text description information can be determined according to actual needs. For example, in the pattern design creation scenario, the terminal device can create a target pattern image that meets the requirements of the text description information based on the original pattern image according to the acquired original pattern image and the corresponding text description information. For another example, when using images for publicity and introduction, the terminal device can generate a target image that meets the publicity information in the text description information based on the instance image according to the acquired instance image and the text description information corresponding to the publicity purpose, so as to improve the publicity effect when the publicity work is officially carried out.
[0115] In this specification, the terminal device can receive an image creation request and obtain the original image containing the specified object and text description information required to create the image according to the image creation request. The text description information has the same function as in the above model training process, and is used to represent the image content of the target image containing the specified object generated based on the original image.
[0116] S503: Inputting the original image and the text description information into an image generation model so that the image generation model generates the target image based on the original image, wherein the image generation model is trained by the above-mentioned model training method.
[0117] In this specification, the terminal device can input the acquired original image and corresponding text description information into the trained image generation model, so that the image generation model generates a target image that conforms to the image content described in the text description information based on the original image.
[0118] It should be noted that in the actual application stage, the image generation model will only generate the target image based on the original image and the corresponding text description information, and will not generate a priori image and then reconstruct the prior image as in the above-mentioned model training stage. Since the model is trained and optimized based on the deviation between the target image and the reference data, and the deviation between the reconstructed prior image and the prior image in the model training stage, the trained image generation model already has the ability to perfectly combine the diversified image features in the prior image generated in the training stage with the original image. Therefore, in actual applications, there is no need for subsequent operations such as generating a priori images based on text description information. The trained image generation model can directly generate a target image that meets the text description information based on the original image according to the various diversified image data learned in the previous training process.
[0119] From the above content, it can be seen that the model training method and image generation method provided in this specification can generate a corresponding prior image based on text description information, and reconstruct the prior image through the image generation model according to the label information of the original image, and then train and optimize the image generation model based on the reconstructed prior image and the prior image.
[0120] The image generation model trained by this method can generate diversified images that not only retain the specified object in the complete original image, but also are highly close to the image content described in the text description information. This effectively improves the image augmentation capability of the image generation model when creating diversified images for the specified object, while fully protecting the integrity of the feature information of the original specified object, greatly improving the fidelity of the specified object in the generated target image and the consistency with the specified object in the original image.
[0121] The above are one or more implementation methods of this specification. Based on the same idea, this specification also provides a corresponding model training device, such as Figure 6 shown.
[0122] Figure 6 A schematic diagram of a model training device provided in this specification, including:
[0123] An acquisition module 601 is used to acquire an original image containing a specified object and text description information, wherein the text description information is used to represent the image content of a target image containing the specified object generated based on the original image;
[0124] A generation module 602, configured to input the original image and the text description information into an image generation model to be trained, so that the image generation model generates the target image based on the original image;
[0125] A reconstruction module 603 is used to input the text description information into the image generation model so that the image generation model generates a priori image corresponding to the text description information, and to determine label information corresponding to the original image, wherein the label information is used to represent the specified object contained in the original image, and to input the label information and the priori image into the image generation model so that the image generation model reconstructs the priori image according to the label information to obtain a reconstructed priori image;
[0126] a deviation determination module 604, configured to determine reference data for the target image, and determine a first deviation based on the reference data and the target image, and determine a second deviation based on the prior image and the reconstructed prior image;
[0127] The training module 605 is used to train the image generation model with minimizing the first deviation and the second deviation as the optimization goal.
[0128] Optionally, the generation module 602 is specifically used to input the original image and the text description information into the image generation model, so that the image generation model determines the original feature data corresponding to the original image based on the original image, and determines the text feature data corresponding to the text description information based on the text description information, and determines the fused feature data for the original image and the text description information based on the original feature data and the text feature data, and generates the target image based on the fused feature data.
[0129] Optionally, the deviation determination module 604 is specifically used to determine original feature data corresponding to the original image according to the original image; determine text feature data corresponding to the text description information according to the text description information; determine reference data for the target image according to the original feature data and the text feature number;
[0130] The deviation determination module 604 is specifically configured to determine, based on the target image, target feature data corresponding to the target image; and determine a deviation between the target feature data and the reference data as the first deviation.
[0131] Optionally, the deviation determination module 604 is specifically used to determine, based on the reconstructed prior image, reconstructed feature data corresponding to the reconstructed prior image; determine, based on the prior image, prior feature data corresponding to the prior image, and, based on the label information, determine label feature data corresponding to the label information; determine, based on the prior feature data and the label feature data, reconstructed reference data for the reconstructed prior image; and determine the second deviation based on the reconstructed feature data and the reconstructed reference data.
[0132] Optionally, the deviation determination module 604 is specifically used to determine the deviation between the reconstructed feature data and the reconstructed reference data as an intermediate deviation; input the prior image and the original image into a preset image similarity calculation model so that the image similarity calculation model determines the similarity value between the prior image and the original image; and determine the second deviation based on the intermediate deviation and the similarity value.
[0133] Based on the same idea, this specification also provides a corresponding image generation device, such as Figure 7 shown.
[0134] Figure 7 A schematic diagram of an image generation device provided in this specification, comprising:
[0135] Receiving module 701, used for receiving an image creation request;
[0136] The acquisition module 702 is used to acquire, according to the image creation request, an original image including a specified object and text description information required for creating an image, wherein the text description information is used to represent the image content of the target image including the specified object generated based on the original image;
[0137] The image generation module 703 is used to input the original image and the text description information into the image generation model so that the image generation model generates the target image based on the original image. The image generation model is trained by the model training method as described above.
[0138] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 and Figure 5 Provides model training methods and image generation methods.
[0139] This manual also provides Figure 8 The schematic structure diagram of the electronic device shown in FIG. Figure 8As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The model training method shown or Figure 5 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0140] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0141] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0142] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0143] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0144] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0145] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0146] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0148] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0149] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0150] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0151] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0152] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0153] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0154] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0155] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.
Claims
1. A model training method, comprising: Acquire an original image containing a specified object and text description information, wherein the text description information is used to represent the image content of a target image containing the specified object generated based on the original image; Inputting the original image and the text description information into an image generation model to be trained, so that the image generation model generates the target image based on the original image; Inputting the text description information into the image generation model so that the image generation model generates a priori image corresponding to the text description information, and determining label information corresponding to the original image, wherein the label information is used to represent the specified object contained in the original image, and inputting the label information and the priori image into the image generation model so that the image generation model reconstructs the priori image according to the label information to obtain a reconstructed priori image; Determine reference data for the target image, and determine a first deviation based on the reference data and the target image, and determine a second deviation based on the prior image and the reconstructed prior image; The image generation model is trained with minimizing the first deviation and the second deviation as an optimization goal.
2. The method according to claim 1, inputting the original image and the text description information into an image generation model to be trained so that the image generation model generates a target image based on the original image, specifically comprising: The original image and the text description information are input into the image generation model, so that the image generation model determines the original feature data corresponding to the original image based on the original image, and determines the text feature data corresponding to the text description information based on the text description information, and determines the fused feature data for the original image and the text description information based on the original feature data and the text feature data, and generates the target image based on the fused feature data.
3. The method according to claim 1, determining reference data for the target image, specifically comprising: Determining original feature data corresponding to the original image according to the original image; Determining text feature data corresponding to the text description information according to the text description information; Determining reference data for the target image according to the original feature data and the text feature number; Determining a first deviation according to the reference data and the target image specifically includes: According to the target image, determining target feature data corresponding to the target image; A deviation between the target feature data and the reference data is determined as the first deviation.
4. The method according to claim 1, determining the second deviation according to the prior image and the reconstructed prior image, specifically comprising: Determining, according to the reconstructed priori image, reconstructed feature data corresponding to the reconstructed priori image; Determine, based on the prior image, prior feature data corresponding to the prior image, and determine, based on the label information, label feature data corresponding to the label information; Determining reconstruction reference data for the reconstructed prior image according to the prior feature data and the label feature data; The second deviation is determined according to the reconstructed feature data and the reconstructed reference data.
5. The method according to claim 4, determining the second deviation according to the reconstructed feature data and the reconstructed reference data, specifically comprising: determining a deviation between the reconstructed feature data and the reconstructed reference data as an intermediate deviation; Inputting the prior image and the original image into a preset image similarity calculation model so that the image similarity calculation model determines a similarity value between the prior image and the original image; The second deviation is determined according to the intermediate deviation and the similarity value.
6. A method for generating an image, comprising: receiving an image creation request; According to the image creation request, an original image including a specified object and text description information required for creating an image are acquired, wherein the text description information is used to represent the image content of a target image including the specified object generated based on the original image; The original image and the text description information are input into an image generation model so that the image generation model generates the target image based on the original image, and the image generation model is trained by the method described in any one of claims 1 to 5 above.
7. A model training device, comprising: An acquisition module, used for acquiring an original image containing a specified object and text description information, wherein the text description information is used for representing the image content of a target image containing the specified object generated based on the original image; A generation module, used for inputting the original image and the text description information into an image generation model to be trained, so that the image generation model generates the target image based on the original image; A reconstruction module, used for inputting the text description information into the image generation model so that the image generation model generates a priori image corresponding to the text description information, and determining label information corresponding to the original image, wherein the label information is used to represent the specified object contained in the original image, and inputting the label information and the priori image into the image generation model so that the image generation model reconstructs the priori image according to the label information to obtain a reconstructed priori image; a deviation determination module, configured to determine reference data for the target image, and determine a first deviation based on the reference data and the target image, and determine a second deviation based on the prior image and the reconstructed prior image; A training module is used to train the image generation model with minimizing the first deviation and the second deviation as an optimization goal.
8. In the device as described in claim 7, the generation module is specifically used to input the original image and the text description information into the image generation model, so that the image generation model determines the original feature data corresponding to the original image based on the original image, and determines the text feature data corresponding to the text description information based on the text description information, and determines the fused feature data for the original image and the text description information based on the original feature data and the text feature data, and generates the target image based on the fused feature data.
9. The device according to claim 7, wherein the deviation determination module is specifically used to determine original feature data corresponding to the original image according to the original image; determine text feature data corresponding to the text description information according to the text description information; and determine reference data for the target image according to the original feature data and the text feature number; The deviation determination module is specifically used to determine, based on the target image, target feature data corresponding to the target image; and determine a deviation between the target feature data and the reference data as the first deviation.
10. In the device as described in claim 7, the deviation determination module is specifically used to determine the reconstructed feature data corresponding to the reconstructed prior image based on the reconstructed prior image; determine the prior feature data corresponding to the prior image based on the prior image, and determine the label feature data corresponding to the label information based on the label information; determine the reconstruction reference data for the reconstructed prior image based on the prior feature data and the label feature data; and determine the second deviation based on the reconstructed feature data and the reconstructed reference data.
11. In the device as described in claim 10, the deviation determination module is specifically used to determine the deviation between the reconstructed feature data and the reconstructed reference data as an intermediate deviation; input the prior image and the original image into a preset image similarity calculation model so that the image similarity calculation model determines the similarity value between the prior image and the original image; and determine the second deviation based on the intermediate deviation and the similarity value.
12. An image generating device, comprising: A receiving module, used for receiving an image creation request; an acquisition module, configured to acquire, according to the image creation request, an original image including a specified object and text description information required for creating an image, wherein the text description information is used to represent the image content of a target image including the specified object generated based on the original image; An image generation module is used to input the original image and the text description information into an image generation model so that the image generation model generates the target image based on the original image, and the image generation model is trained by the method described in any one of claims 1 to 5 above.
13. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 6.
14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the program.
