Model training method and image generation method

Abstract

The invention provides a model training method and an image generation method, which can be applied to the technical field of artificial intelligence and computer vision. The model training method comprises the following steps: based on target sample attribute information, performing image transformation on a sample object in an original sample image by using a first generator to generate a target sample image; performing image transformation on a sample object in the target sample image based on the original attribute information by using a second generator to generate a restored sample image; determining a first loss value based on the feature similarity between the original sample image and the target sample image; performing similarity discrimination on the original sample image and the target sample image by using a first discriminator matched with the first generator, and determining a second loss value; determining a third loss value based on the feature similarity between the original sample image and the restored sample image; and training the first generator based on the first loss value, the second loss value and the third loss value to obtain a trained first generator.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and computer vision technology, and in particular to a model training method and an image generation method. Background Art

[0002] With the development of artificial intelligence and computer vision technologies, facial recognition, as a key biometric technology, has been widely used in security surveillance, financial payments, smart homes, and other scenarios. This technology extracts characteristic information from facial images for identity verification, offering advantages such as contactless, convenient, and efficient operation. In some facial recognition systems designed for long-term use, users' biometrics may change significantly with age, resulting in reduced recognition accuracy. Requiring users to regularly upload their latest images and update the model can degrade the user experience, and incremental updates during model operation can affect model stability.

[0003] Using artificial intelligence to generate faces across ages can achieve identity-consistent face editing by simulating age changes. Based on a small number of user portraits, the user's biometric characteristics at different age groups can be predicted and different portraits can be generated. The generated portraits of users at different age groups are used as data sets to train the face recognition model, thus completing the data set expansion without the need for users to upload real images on their own.

[0004] In the process of implementing the present disclosure, the inventors discovered that the prior art has at least the following problems: when updating the original image based on the changed object attribute information, there is a problem of deviation in the identity of the object in the updated image. Summary of the Invention

[0005] In view of the above problems, the present disclosure provides a model training method and an image generation method.

[0006] According to a first aspect of the present disclosure, a model training method is provided, comprising: performing image transformation on a sample object in an original sample image using a first generator based on target sample attribute information to generate a target sample image, wherein the target sample attribute information is different from the original attribute information of the sample object in the original sample image; performing image transformation on the sample object in the target sample image based on the original attribute information to generate a restored sample image; determining a first loss value based on feature similarity between the original sample image and the target sample image; performing similarity discrimination on the original sample image and the target sample image using a first discriminator matched with the first generator to determine a second loss value; determining a third loss value based on feature similarity between the original sample image and the restored sample image; and training the first generator based on the first loss value, the second loss value, and the third loss value to obtain a trained first generator.

[0007] According to an embodiment of the present disclosure, the model training method also includes: determining a fourth loss value based on the feature similarity between the target sample image and the restored sample image; using a second discriminator matched with the second generator to perform similarity judgment on the target sample image and the restored sample image to determine a fifth loss value; determining a sixth loss value based on the feature similarity between the original sample image and the restored sample image; training the second generator based on the fourth loss value, the fifth loss value and the sixth loss value to obtain a trained second generator.

[0008] According to an embodiment of the present disclosure, the first generator is trained based on the first loss value, the second loss value and the third loss value, including: when it is determined that the trained first generator does not meet the training conditions, the model parameters of the trained second generator and the trained first generator are exchanged to obtain a new first generator and a new second generator; using the new first generator and the new second generator, re-training is performed based on the original image, the target sample attribute information and the original attribute information to obtain a new trained first generator.

[0009] According to an embodiment of the present disclosure, based on the target sample attribute information, a first generator is used to perform image transformation on the sample objects in the original sample image to generate the target sample image, including: using the first generator to perform the following operations: determining a feature change parameter based on a first average object feature corresponding to the original sample attribute information and a second average object feature corresponding to the target sample attribute information, wherein the first average object feature is determined by using the object features of each of the multiple objects corresponding to the original sample attribute information, and the second average object feature is determined by using the object features of each of the multiple objects corresponding to the target sample attribute information; and transforming the object features of the sample objects in the original sample image according to the feature change parameter to generate the target sample image.

[0010] According to an embodiment of the present disclosure, the average object feature includes a first average object feature or a second average object feature; the average object feature is determined as follows: based on multiple target reference images that match preset attribute information, determining the reference features of the objects included in each of the multiple target reference images; and determining the average object feature of the preset attribute information based on the average value of the multiple reference features.

[0011] According to an embodiment of the present disclosure, the first average object feature includes multiple first average feature points, and the second average object feature includes multiple second average feature points, wherein the first average feature points and the second average feature points correspond one to one; based on the first average object feature corresponding to the original sample attribute information and the second average object feature corresponding to the target sample attribute information, the feature change parameter is determined, including: for each pair of mutually corresponding first average feature points and second average feature points, determining the position deviation between the first average feature point and the second average feature point; and determining the feature change parameter between the first average feature point and the second average feature point based on the position deviation.

[0012] According to an embodiment of the present disclosure, the feature change parameters include a feature change direction and a feature change amplitude; according to the feature change parameters, the object features of the sample object in the original sample image are transformed to generate a target sample image, including: according to the feature change direction and the feature change amplitude, the object features of the sample object are translated to generate a target sample image.

[0013] The second aspect of the present disclosure provides an image generation method, comprising: generating a target image using a target generator based on target attribute information and original attribute information of a target object in an original image, wherein the target generator is trained according to the steps of the above-mentioned model training method.

[0014] According to an embodiment of the present disclosure, the image generation method also includes: generating an initial guide text based on target attribute information; embedding the image features of the target image into the text features of the initial guide text to obtain a rendering code; and rendering the target image according to the rendering code to obtain a rendered target image.

[0015] According to an embodiment of the present disclosure, the image features of the target image are embedded in the text features of the initial guide text to obtain a rendering code, including: determining the image code of the target image and the text code of the initial guide text; and superimposing the image code and the text code to obtain a rendering code.

[0016] The third aspect of the present disclosure provides a model training device, including: a first image transformation module, which is used to use a first generator to perform image transformation on a sample object in an original sample image based on target sample attribute information to generate a target sample image, wherein the target sample attribute information is different from the original attribute information of the sample object in the original sample image; a second image transformation module, which is used to use a second generator to perform image transformation on a sample object in a target sample image based on the original attribute information to generate a restored sample image; a first loss determination module, which is used to determine a first loss value based on feature similarity between the original sample image and the target sample image; a second loss determination module, which is used to use a first discriminator matched with the first generator to perform similarity discrimination on the original sample image and the target sample image to determine a second loss value; a third loss determination module, which is used to determine a third loss value based on feature similarity between the original sample image and the restored sample image; a first model training module, which is used to train the first generator based on the first loss value, the second loss value and the third loss value to obtain a trained first generator.

[0017] The fourth aspect of the present disclosure provides an image generation device, including: an image generation module, used to generate a target image based on target attribute information and original attribute information of the target object in the original image, using a target generator, wherein the target generator is trained using the above-mentioned model training device.

[0018] A fifth aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0019] The sixth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instruction stored thereon, which implements the steps of the above method when the computer program or instruction is executed by a processor.

[0020] The seventh aspect of the present disclosure further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0021] According to an embodiment of the present disclosure, a sample image in an original sample image is transformed using target sample attribute information to obtain a target sample image. A second generator then performs image transformation on the sample image in the target sample image, restoring the object features of the sample object to the original attribute information to obtain a restored sample image. A total loss for a first generator is determined based on a combination of multiple loss values, and the first generator is trained based on the total loss to obtain a trained first generator. In addition to the cosine loss between the images before and after processing by the first generator and the generation loss of the first generator itself, the loss also includes a cycle consistency loss between the original sample image and the restored sample image. This means that parameter adjustments to the first generator are not limited to the processing of the first generator itself but also involve the restored sample image after processing by the second generator. This ensures that the trained first generator has a strong and flexible ability to adapt to changes in sample attribute information, thereby improving the generation capability of the trained first generator. Furthermore, by comparing the cycle consistency loss between the original sample image and the restored sample image, the processing of the first generator is made reversible, ensuring that the sample objects in the target sample image obtained by the first generator still possess the original object features of the sample objects, thereby avoiding the problem of object identity deviation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0023] Figure 1 Schematically illustrates an application scenario diagram of a model training method and an image generation method, apparatus, device, medium, and program product according to an embodiment of the present disclosure;

[0024] Figure 2 The following schematically shows a flow chart of a model training method according to an embodiment of the present disclosure;

[0025] Figure 3 Schematically shows a structural diagram of model training according to an embodiment of the present disclosure;

[0026] Figure 4 The structure diagram of the generator according to the embodiment of the present disclosure is schematically shown;

[0027] Figure 5 Schematically shows a structural diagram of a discriminator according to an embodiment of the present disclosure;

[0028] Figure 6 A diagram schematically illustrates a comparison of object features of different attribute information according to an embodiment of the present disclosure;

[0029] Figure 7 The flowchart of the image generation method according to the embodiment of the present disclosure is schematically shown;

[0030] Figure 8 A structural diagram schematically illustrates an image rendering method according to an embodiment of the present disclosure;

[0031] Figure 9 The following schematically shows a structural block diagram of a model training device according to an embodiment of the present disclosure;

[0032] Figure 10 A block diagram schematically illustrates a structure of an image generating device according to an embodiment of the present disclosure; and

[0033] Figure 11 A block diagram of an electronic device suitable for implementing a model training method and an image generation method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0035] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0036] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0037] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0038] In the technical solutions disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0039] In scenarios where personal information is used for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure all provide users with corresponding operation portals for them to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge, and skills, and have reached a certain level of professionalism.

[0040] An embodiment of the present disclosure provides a model training method, including: based on target sample attribute information, using a first generator to perform image transformation on a sample object in an original sample image to generate a target sample image, wherein the target sample attribute information is different from the original attribute information of the sample object in the original sample image; using a second generator to perform image transformation on the sample object in the target sample image based on the original attribute information to generate a restored sample image; based on the feature similarity between the original sample image and the target sample image, determining a first loss value; using a first discriminator matched with the first generator to perform similarity discrimination on the original sample image and the target sample image to determine a second loss value; based on the feature similarity between the original sample image and the restored sample image, determining a third loss value; and training the first generator based on the first loss value, the second loss value, and the third loss value to obtain a trained first generator.

[0041] Figure 1 The application scenario diagram of the model training method and image generation method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0042] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0043] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0044] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0045] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0046] It should be noted that the model training method and image generation method provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the model training device and image generation device provided in the embodiments of the present disclosure can generally be set in the server 105. The model training method and image generation method provided in the embodiments of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the model training device and image generation device provided in the embodiments of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0047] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0048] Figure 2 The flowchart of the model training method according to an embodiment of the present disclosure is schematically shown.

[0049] like Figure 2 As shown, the model training method of this embodiment includes operations S210 to S260.

[0050] In operation S210 , based on the target sample attribute information, a first generator is used to perform image transformation on the sample object in the original sample image to generate a target sample image.

[0051] According to an embodiment of the present disclosure, the target sample attribute information is different from the original attribute information of the sample object in the original sample image. The target sample attribute information and the original attribute information may include object attributes such as the age of the sample object. The original sample image may be parsed to determine the age of the sample object, thereby determining the original attribute information. The target sample attribute information may be used to represent the attribute information of the sample object in the target sample image after the image transformation, that is, the sample object in the target sample image has the same identity information as the sample object in the original sample image, but the attribute information of the sample objects of the two is different. According to an embodiment of the present disclosure, the first generator may be used to perform image transformation on the sample object with the original attribute information in the sample image according to the target sample attribute information, thereby generating a sample object with the target sample attribute information.

[0052] For example, the original attribute information of the sample object in the original sample image is age, 20 years old. The target sample attribute information is age 30 years old. The image transformation is performed on the sample object of 20 years old in the original sample image to generate a target sample image of the sample object at the age of 30.

[0053] In operation S220, a second generator is used to perform image transformation on the sample object in the target sample image based on the original attribute information to generate a restored sample image.

[0054] According to an embodiment of the present disclosure, the second generator can be used to process the target sample image with target sample attribute information generated by the first generator to obtain a restored sample image, wherein the sample object in the restored sample image has the original attribute information.

[0055] According to the embodiments of the present disclosure, since both the restored sample image and the original sample image are images of sample objects with original attribute information, ideally, when both the first generator and the second generator meet the generation capabilities required, the restored sample image and the original sample image have high similarity and small differences. However, since both the first generator and the second generator are untrained image generators, in addition to the attribute information of the sample objects in the original sample image being original attribute information, there are differences between the attribute information of the sample objects in the target sample image and the target sample attribute information, as well as between the attribute information of the sample objects in the restored sample image and the original attribute information, resulting in differences between the restored sample image and the original sample image.

[0056] In operation S230 , a first loss value is determined based on feature similarity between the original sample image and the target sample image.

[0057] According to an embodiment of the present disclosure, the feature similarity between the original sample image and the target sample image can be expressed by the cosine loss between the two, wherein the original sample image and the target sample image can be represented in the feature space respectively according to the feature point cloud of the sample object in the original sample image and the feature point cloud of the sample object in the target sample image, and the cosine loss of the above two images in the feature space is calculated to determine the first loss value.

[0058] In operation S240 , a first discriminator matched with the first generator is used to perform similarity discrimination on the original sample image and the target sample image to determine a second loss value.

[0059] According to an embodiment of the present disclosure, the first discriminator can judge the target sample image based on the original sample image, determine the probability that it is a real sample image, take the logarithm of the probability, and calculate the expectation based on the sampling data obtained by sampling in the real sample image to obtain the second loss value, as shown in formula (1):

[0060] (1)

[0061] Among them, L D1 represents the second loss value, E(·) is the expectation, D1 represents the first discriminator, p data Represents the distribution of real sample images, x~p data Represents the sample data x sampled from the real sample image, G1(x) represents the target sample image output by the first generator, and D1(G1(x)) represents the probability that the target sample image is real data.

[0062] In operation S250 , a third loss value is determined based on feature similarity between the original sample image and the restored sample image.

[0063] According to an embodiment of the present disclosure, the third loss value can be determined using the cycle consistency loss between the original sample image and the restored sample image, wherein the cycle consistency loss can be used to measure the reversibility of the conversion process from the original sample image to the restored sample image.

[0064] In operation S260, the first generator is trained based on the first loss value, the second loss value, and the third loss value to obtain a trained first generator.

[0065] According to an embodiment of the present disclosure, based on the first loss value, the second loss value, and the third loss value, the total loss of the first generator can be determined, and the total loss of the first generator is used to adjust the parameters of the first generator to obtain the trained first generator. The total loss of the first generator is shown in formula (2):

[0066] (2)

[0067] Among them, α, β, and γ are weights. Preferably, α=1.2, β=1, and γ=1.5 can be taken. L cos1 is the first loss value, is the third loss value, that is, the cycle consistency loss between the original sample image and the restored sample image. The parameter list of this loss value is (G1, G2), which means that the original sample image is first processed by the first generator, and then the target sample image obtained by the first generator is processed by the second generator to obtain the restored sample image, and the cycle consistency loss between the original sample image and the restored sample image is judged.

[0068] According to an embodiment of the present disclosure, a sample image in an original sample image is transformed using target sample attribute information to obtain a target sample image. A second generator then performs image transformation on the sample image in the target sample image, restoring the object features of the sample object to the original attribute information to obtain a restored sample image. A total loss for a first generator is determined based on a combination of multiple loss values, and the first generator is trained based on the total loss to obtain a trained first generator. In addition to the cosine loss between the images before and after processing by the first generator and the generation loss of the first generator itself, the loss also includes a cycle consistency loss between the original sample image and the restored sample image. This means that parameter adjustments to the first generator are not limited to the processing of the first generator itself but also involve the restored sample image after processing by the second generator. This ensures that the trained first generator has a strong and flexible ability to adapt to changes in sample attribute information, thereby improving the generation capability of the trained first generator. Furthermore, by comparing the cycle consistency loss between the original sample image and the restored sample image, the processing of the first generator is made reversible, ensuring that the sample objects in the target sample image obtained by the first generator still possess the original object features of the sample objects, thereby avoiding the problem of object identity deviation.

[0069] According to an embodiment of the present disclosure, the model training method also includes: determining a fourth loss value based on the feature similarity between the target sample image and the restored sample image; using a second discriminator matched with the second generator to perform similarity judgment on the target sample image and the restored sample image to determine a fifth loss value; determining a sixth loss value based on the feature similarity between the original sample image and the restored sample image; training the second generator based on the fourth loss value, the fifth loss value and the sixth loss value to obtain a trained second generator.

[0070] According to an embodiment of the present disclosure, similar to the training process of the first generator, the cosine loss between the target sample image and the restored sample image can be used to characterize the feature similarity between the two, that is, the fourth loss value.

[0071] According to an embodiment of the present disclosure, the second discriminator can judge the restored sample image based on the target sample image, determine the probability that it is real data, take the logarithm of the probability, and calculate the expectation based on the random noise data to obtain the fifth loss value, as shown in formula (3):

[0072] (3)

[0073] Among them, L D2 represents the fifth loss value, D2 represents the second discriminator, p z Represents the distribution of random noise data, x~p zRepresents x sampled from the real sample, G1(x) represents the restored sample image output by the second generator, and D2(G2(x)) represents the probability that the restored sample image is the real data.

[0074] According to an embodiment of the present disclosure, the sixth loss value may be determined using the cycle consistency loss between the restored sample image and the original sample image.

[0075] In another embodiment, the second generator can be used to process the original sample image to obtain an intermediate sample image, and then the first generator can be used to process the intermediate sample image to obtain another new restored sample image. Based on the cycle consistency loss between the new restored sample image and the original sample image, the sixth loss value can be determined.

[0076] According to an embodiment of the present disclosure, since the third loss value and the sixth loss value are both cycle consistency losses, but the conversion paths of the third loss value and the sixth loss value are different, that is, the third loss value is the cycle consistency loss between the original sample image and the restored sample image, and the sixth loss value is the cycle consistency loss between the restored sample image and the original sample.

[0077] According to an embodiment of the present disclosure, the first generator is trained based on the first loss value, the second loss value and the third loss value, including: when it is determined that the trained first generator does not meet the training conditions, the model parameters of the trained second generator and the trained first generator are exchanged to obtain a new first generator and a new second generator; using the new first generator and the new second generator, re-training is performed based on the original image, the target sample attribute information and the original attribute information to obtain a new trained first generator.

[0078] According to the embodiments of the present disclosure, during the model training process, only the original sample images are real data; the remaining data, such as the target sample images, are processed by the generator, and their authenticity is difficult to guarantee. Therefore, if only the target sample images are used to train the second generator, it is difficult to ensure the second generator's generation capability, and thus the authenticity of the restored sample images cannot be guaranteed. This affects the third loss value determined based on the restored sample images, thereby affecting the training process of the first generator and resulting in low training accuracy for the first generator.

[0079] To address the above issues, according to an embodiment of the present disclosure, the parameters of the first generator are adjusted based on the total loss of the first generator to obtain the trained first generator. The total loss of the second generator can be determined based on the fourth loss value, the fifth loss value, and the sixth loss value. The parameters of the second generator are adjusted based on the total loss of the second generator to obtain the trained second generator.

[0080] According to an embodiment of the present disclosure, the total loss of the second generator is shown in formula (4):

[0081] (4)

[0082] Among them, L cos2 is the fourth loss value, is the sixth loss value. Since the parameter list of this loss value is (G2, G1), it means that the second generator is used to process the original sample image first, and then the first generator is used to process the intermediate sample image obtained by the second generator to obtain another new restored sample image, and the cycle consistency loss between the new restored sample image and the original sample image is determined.

[0083] According to an embodiment of the present disclosure, the original sample image is processed using the trained first generator and the trained second generator according to the above process to obtain a new target sample image and a new restored sample image, respectively. According to the above model training method, the total loss of the trained first generator is calculated. When it is determined that the total loss of the first generator has not converged, it can be determined that the trained first generator does not meet the training conditions.

[0084] According to an embodiment of the present disclosure, when it is determined that the first generator after training does not meet the training conditions, a new second generator is determined based on the model parameters of the trained first generator, and a new first generator is determined based on the model parameters of the trained second generator. The above process is repeated using the new first generator and the new second generator to obtain a new trained first generator.

[0085] According to an embodiment of the present disclosure, after each new trained first generator, it is determined whether the trained first generator meets the training conditions. If the training conditions are not met, the above training process is repeated until the newly obtained trained first generator meets the training conditions. The current trained first generator is determined as the trained first generator, and the training of the first generator is completed.

[0086] According to an embodiment of the present disclosure, after each training, the parameters of the trained first generator and the trained second generator are exchanged, so that the model parameters of each generator can be calculated using real data, thereby improving the training accuracy of the first generator and the second generator.

[0087] Figure 3 The structural diagram of the model training according to an embodiment of the present disclosure is schematically shown.

[0088] like Figure 3As shown in the figure, the model structure includes a first generator G1, a first discriminator D1 corresponding to the first generator G1, a second generator G2, and a second discriminator D2 corresponding to the second generator G2. During the training process, the first generator G1 is used to process the original sample image to generate the target sample image, and the second generator G2 is used to process the target sample image to generate the restored sample image.

[0089] According to the original sample image and the target sample image, the cosine loss is calculated, and the generation loss between the original sample image and the target sample image is determined by using the first discriminator D1. Then, according to the target sample image and the restored sample image, the cycle consistency loss between the two is determined. According to the above three losses, the total loss of the first generator is determined, and the parameters of the first generator are adjusted. A similar method is used to adjust the parameters of the second generator. After the parameter adjustment is completed, the parameters of the trained first generator and the trained second generator are swapped, that is, the parameters of the trained second generator are used as the new first generator, and the parameters of the trained first generator are used as the new second generator. The above parameter adjustment process is repeated until the adjusted first generator converges, and the parameters of the current adjusted first generator are determined, and it is used as the target generator in subsequent applications.

[0090] Figure 4 The figure schematically shows a structural diagram of a generator according to an embodiment of the present disclosure.

[0091] like Figure 4 As shown, taking the first generator as an example, the object features are input into the generator, and after being processed by the linear layer and the nonlinear layer, feature modulation is performed to enhance the model's utilization of features and improve the model's flexibility and interpretability, wherein the feature modulation can be performed using the FiLM layer. Preferably, 7 layers of FiLM layers can be set. The original sample attribute information and the target sample attribute information are embedded separately, and feature fusion is performed with the features after feature modulation. The fused features are processed by the linear layer-nonlinear layer-linear layer to obtain the predicted object features. According to the predicted object features, the sample objects in the original sample image can be adjusted to generate the target sample image, wherein the predicted object features can include the object features of the sample object itself and the features of the target sample attribute information after the attribute information is changed.

[0092] The object features can be 512 groups of 81 feature points in the (512, 162) dimension. However, for each batch size of 512 groups of data, each group has only a single-bit label, which is insufficient to influence the subsequent generation process. Therefore, the generator adopts a dual-path embedding strategy, embedding the original sample attribute information and the target sample attribute information separately, concatenating them and injecting them into the deep network. Specifically, the 1-bit label can be expanded to 10 bits, and then the expanded label can be concatenated into the feature-modulated feature to complete feature fusion. The fused features are processed through a linear layer, a nonlinear layer, and a linear layer until the output dimension is adjusted to (512, 162). The resulting 512 groups of 81 feature points in the (512, 162) dimension are determined as the predicted object features.

[0093] Figure 5 The figure schematically shows a structural diagram of a discriminator according to an embodiment of the present disclosure.

[0094] like Figure 5 As shown, taking the second generator as an example, the discriminator can embed the target sample attribute information and fuse it with the original sample attribute information. After being processed by the linear layer, it is processed in the form of alternating nonlinear layer-linear layer, and the activation function is used to output the confidence to obtain the second loss value.

[0095] The discriminator's embedding layer maps attribute information labels into 10-dimensional continuous vectors, avoiding the information bottleneck of scalar labels and enabling the model to capture nonlinear differences in the distribution of attribute information. By alternating nonlinear and linear layers, high-order feature interactions are gradually extracted through cascading nonlinear transformations. Finally, activation functions are used to perform dimensionality compression and achieve hierarchical dimensionality reduction, reducing computational complexity while preserving the discriminator's accuracy.

[0096] According to an embodiment of the present disclosure, based on the target sample attribute information, a first generator is used to perform image transformation on the sample objects in the original sample image to generate the target sample image, including: using the first generator to perform the following operations: determining a feature change parameter based on a first average object feature corresponding to the original sample attribute information and a second average object feature corresponding to the target sample attribute information, wherein the first average object feature is determined by using the object features of each of the multiple objects corresponding to the original sample attribute information, and the second average object feature is determined by using the object features of each of the multiple objects corresponding to the target sample attribute information; and transforming the object features of the sample objects in the original sample image according to the feature change parameter to generate the target sample image.

[0097] According to an embodiment of the present disclosure, the first average object feature can be used to represent the average value of the object features of the objects having the original sample attribute information, and the second average object feature can be used to represent the average value of the object features of the objects having the target sample attribute information. The object features can be represented by the coordinates of each feature point in the object's contour feature point cloud. For example, if the original sample attribute information is 15 years old, the average feature point cloud can be calculated based on the facial contour feature point clouds of multiple 15-year-old objects to determine the first average object feature.

[0098] According to an embodiment of the present disclosure, the target sample attribute information and the original sample attribute information can also be set as value intervals. For example, the attribute information intervals can be divided into seven intervals: 0-9 years old, 10-19 years old, 20-29 years old, 30-39 years old, 40-49 years old, 50-69 years old, and over 70 years old. In this case, if the original sample attribute information is 15 years old, the average feature point cloud of 10-19 years old can be used as the first average object feature.

[0099] According to the embodiments of the present disclosure, the coordinates of the average feature point group are extracted as the average object features of different attribute information, which can obtain more universal object features, facilitate eliminating the influence of age groups during training, and make the verification of identity features more accurate.

[0100] According to an embodiment of the present disclosure, the average feature point coordinates are determined as follows: based on multiple target reference images that match the preset attribute information, the reference features of the objects included in each of the multiple target reference images are determined; and based on the average value of the multiple reference features, the average feature point coordinates of the preset attribute information are determined.

[0101] According to the embodiments of the present disclosure, a large number of reference images including different objects can be obtained by means of a public data set. By analyzing the multiple reference images separately, the attribute information of the object in each reference image can be determined.

[0102] After determining multiple reference features, using the extraction of 81 feature points as an example, corresponding feature points can be identified from each of the reference features, and the average of these corresponding feature points can be calculated to determine the feature point in the average object feature. For example, for the bottommost feature point among the 81 feature points, the coordinates of the bottommost feature points of each of the reference features can be determined. The coordinates of these feature points can be averaged to obtain the coordinates of the bottommost feature point in the average object feature.

[0103] According to the embodiments of the present disclosure, since the coordinates of feature points can be used to represent the outline of an object, that is, the object features, different object features are obtained for reference images with different initial poses of the object. The reference images can be preprocessed so that the initial position and initial angle of the object in the reference images are roughly the same, thereby avoiding deviations caused by the different initial poses of the objects in the reference images. Preprocessing can include translating and rotating the objects in the reference images. After obtaining the average feature point group through this measure, it can be used to calculate the first loss and the fourth loss in the generator training.

[0104] According to an embodiment of the present disclosure, the contour feature point cloud of an object can be determined based on the facial contour of the object. On the basis of 68-point facial landmarks, 13 additional feature points can be collected on the forehead to obtain 81 feature points. The 81 feature points in the contour feature point cloud of the object are used as object features.

[0105] Figure 6 A comparison diagram schematically illustrates object features of different attribute information according to an embodiment of the present disclosure.

[0106] like Figure 6 As shown, the object features include 81 groups of feature points, among which the feature points in the dotted box are the feature points additionally collected on the forehead part of the present disclosure, and the feature points outside the dotted box are the 68 commonly used facial landmark points. Each group of feature points includes three feature points with corresponding relationships. In each group of feature points, red represents the feature point position of the object feature of 0-9 years old at the point, green represents the feature point position of the object feature of 10-19 years old at the point, and blue represents the feature point position of the object feature of 20-29 years old at the point. Since only the first three age groups have obvious contour changes, only the average feature point groups of the first three groups can be calculated.

[0107] According to an embodiment of the present disclosure, the feature change parameter may include the feature point change of each feature point in a plurality of feature points. For example, when the original sample attribute information is in the range of 10 to 19 years old and the target sample attribute information is in the range of 20 to 29 years old, the feature change parameter may be calculated based on the feature point change of each feature point. Figure 6 For each green feature point and its corresponding blue feature point, calculate the position change between the two feature points as the feature point change of the feature point.

[0108] According to an embodiment of the present disclosure, after determining the feature change, the object features of the sample object in the original sample image can be transformed according to the change of each feature point. That is, each feature point in the object features is transformed according to the feature point change corresponding to the feature point, thereby obtaining a target sample image. The transformation of the feature points may include translation.

[0109] According to the embodiments of the present disclosure, a more universal object feature can be obtained based on the first average object feature and the second average object feature. Based on the difference between the two sets of average object features, a feature change parameter is determined, and the average object feature change when the age range corresponding to the original sample attribute information changes to the age range corresponding to the target sample attribute information can be obtained. Because the object in the target sample image is obtained by transforming the object features of the sample object based on this change, the transformed object still has the characteristics of the sample object itself, and using the average object feature change to predict it can improve the authenticity of the object in the target sample image.

[0110] According to an embodiment of the present disclosure, the average object feature includes a first average object feature or a second average object feature; the average object feature is determined as follows: based on multiple target reference images that match preset attribute information, determining the reference features of the objects included in each of the multiple target reference images; and determining the average object feature of the preset attribute information based on the average value of the multiple reference features.

[0111] According to the embodiments of the present disclosure, a large number of reference images including different objects can be obtained by means of a public data set. By analyzing the multiple reference images separately, the attribute information of the object in each reference image can be determined.

[0112] According to an embodiment of the present disclosure, taking the division of attribute information intervals into seven intervals, 0-9 years old, 10-19 years old, 20-29 years old, 30-39 years old, 40-49 years old, 50-69 years old, and over 70 years old, as an example, multiple reference images can be divided into different attribute information intervals according to the attribute information of the objects included in the reference images. Each preset attribute information can correspond to an attribute information interval. For each preset attribute information, multiple target reference images within the attribute information interval are selected, and reference features of the objects included in each target reference image are determined.

[0113] After determining multiple reference features, using the extraction of 81 feature points as an example, corresponding feature points can be identified from each of the reference features, and the average of these corresponding feature points can be calculated to determine the feature point in the average object feature. For example, for the bottommost feature point among the 81 feature points, the coordinates of the bottommost feature points of each of the reference features can be determined. The coordinates of these feature points can be averaged to obtain the coordinates of the bottommost feature point in the average object feature.

[0114] According to the embodiments of the present disclosure, since the coordinates of feature points can be used to represent the outline of an object, that is, the object features, different object features are obtained for reference images with different initial postures of the object. The reference image can be preprocessed so that the initial position and initial angle of the object in the reference image are roughly the same, thereby avoiding deviations caused by the different initial postures of the object in the reference image. Preprocessing can include translating and rotating the object in the reference image. Similarly, the same operations can be performed on the original sample image to make the target sample image and the restored sample image obtained after the image transformation more standard and easy to read.

[0115] According to an embodiment of the present disclosure, the first average object feature includes multiple first average feature points, and the second average object feature includes multiple second average feature points, wherein the first average feature points and the second average feature points correspond one to one; based on the first average object feature corresponding to the original sample attribute information and the second average object feature corresponding to the target sample attribute information, the feature change parameter is determined, including: for each pair of mutually corresponding first average feature points and second average feature points, determining the position deviation between the first average feature point and the second average feature point; and determining the feature change parameter between the first average feature point and the second average feature point based on the position deviation.

[0116] According to an embodiment of the present disclosure, the correspondence between the first average feature point and the second average feature point can be determined as follows: Figure 6 The object feature comparison diagram shown confirms that there is a corresponding relationship between feature points with the same relative position in the feature point clouds of the respective object features.

[0117] According to an embodiment of the present disclosure, for each pair of corresponding first and second average feature points, the coordinate difference between the first and second average feature points can be calculated to obtain a positional deviation, where the positional deviation can be expressed in the form of a coordinate pair. Because the positional deviation can be used to indicate the direction and amount of change required to change from the first average feature point to the second average feature point, the feature change parameter can be determined based on the positional deviation.

[0118] According to an embodiment of the present disclosure, during the generation process, the contour change of the object is decomposed into the change of each feature point on the contour, which can simplify the calculation process, ensure the calculation accuracy, and thus improve the authenticity of the object after the change.

[0119] According to an embodiment of the present disclosure, the feature change parameters include a feature change direction and a feature change amplitude; according to the feature change parameters, the object features of the sample object in the original sample image are transformed to generate a target sample image, including: according to the feature change direction and the feature change amplitude, the object features of the sample object are translated to generate a target sample image.

[0120] According to the embodiments of the present disclosure, since the feature change parameter for each feature point can be the positional deviation determined based on the first average feature point and the second average feature point, and the positional deviation can be expressed as a coordinate pair, the feature change direction and feature change magnitude can be determined based on the coordinate pair.

[0121] According to embodiments of the present disclosure, a coordinate pair can be represented as a vector, where the direction and magnitude of the vector represent the direction and magnitude of feature change, respectively. A translation process can be performed based on the vector to obtain translated feature points. Based on multiple translated feature points, a transformed feature point cloud can be determined. Based on this feature point cloud, a sample object can be adaptively transformed and rendered to obtain a target sample image. The transformation and rendering can include stretching and compression.

[0122] According to the embodiments of the present disclosure, the translation of feature points in the object features is determined based on the feature change direction and feature change amplitude, thereby completing the adjustment of the object features. Because the adjustment is performed based on the sample objects in the original sample image, it is possible to ensure that the adjusted sample objects still have the object features of the sample objects in the original sample image. Because the adjustment process is determined based on the changes between the first average feature points and the second average feature points determined from a large number of samples, it is possible to ensure that the change process is more consistent with the general change law, thereby improving the authenticity of the changed object features.

[0123] According to an embodiment of the present disclosure, the model training method may further include: generating a sample guide text based on the target sample attribute information; embedding the image features of the target sample image into the text features of the sample guide text to obtain a sample rendering code; and rendering the target sample image according to the sample rendering code to obtain a rendered target sample image.

[0124] According to an embodiment of the present disclosure, during the training process, the rendering loss in the rendering process can be determined based on the rendered target sample image and the real image of the sample object with the target sample attribute information, and fine-tuning can be performed based on the rendering loss.

[0125] Specifically, fine-tuning based on rendering loss can be divided into two stages. In the first stage of fine-tuning, the model parameters are fixed and the sample rendering code is fine-tuned until the preset rounds are reached. This completes the first stage of fine-tuning and produces the fine-tuned sample rendering code. In the second stage of fine-tuning, the fine-tuned sample rendering code remains unchanged and the model parameters are fine-tuned until the preset rounds are reached. This completes the second stage of fine-tuning and produces the fine-tuned model.

[0126] Figure 7The flowchart of the image generating method according to the embodiment of the present disclosure is schematically shown.

[0127] like Figure 7 As shown, the image generating method of this embodiment includes operation S710.

[0128] In operation S710, a target image is generated using a target generator based on target attribute information and original attribute information of a target object in an original image.

[0129] According to an embodiment of the present disclosure, a target generator is trained according to the steps of the above-mentioned model training method. Using the target generator, the target object in the original image can be transformed to obtain a target object with target attribute information, and a target image can be determined. Before using the target generator to perform image transformation on the original image, the original image can also be standardized through preprocessing, and the target object can be translated and rotated to adjust the initial position and initial angle of the object, ensuring that the position and angle of the object in the further generated target image are the standardized position and angle, thereby improving the readability of the target image.

[0130] According to an embodiment of the present disclosure, since the target generator has been trained using a model training method, the target generator can more accurately transform the target object in the original image into a target object with target information, thereby ensuring the accuracy of the target image.

[0131] According to an embodiment of the present disclosure, the image generation method also includes: generating an initial guide text based on target attribute information; embedding the image features of the target image into the text features of the initial guide text to obtain a rendering code; and rendering the target image according to the rendering code to obtain a rendered target image.

[0132] According to the embodiments of the present disclosure, since the target image is obtained by simply scaling the outline of the target object in the original image, although it is possible to ensure that the outline of the target object in the transformed target image is correct, it is difficult to ensure that the texture of the target object in the image is normal.

[0133] According to an embodiment of the present disclosure, by setting the rendering code, the rendering code and the target image are input into a large language model for image processing, and the target image is further rendered and fine-tuned to ensure that the rendered target image is more realistic and improve the accuracy of image generation.

[0134] According to an embodiment of the present disclosure, since the target object in the rendered target image needs to have the individual characteristics of the target object and also needs to have the changed target attribute information, an initial guidance text can be generated based on the target attribute information. In addition to the target attribute information, the initial guidance text can also include some quantifiable individual characteristics of the target object, such as gender, etc. The initial guidance text can be text in various languages, as long as it can express the individual characteristics of the target object. Taking the initial guidance text in Chinese as an example, the initial guidance text can be: She is a young woman in her twenties, He is a middle-aged man in his forties, etc. In another example, the initial guidance text in English that expresses the same meaning as the above-mentioned initial guidance text in Chinese can be: She is a young woman in her twenties, He is an amiddle-aged man in his forties.

[0135] According to the embodiments of the present disclosure, in addition to quantifiable individual characteristics such as gender, the target object may also have a large number of difficult-to-quantify individual characteristics, such as cheekbone height, nose bridge height, and eye distance. These individual characteristics can all be reflected in the image features of the target image. Therefore, the image features of the target image can be embedded in the initial guide text to obtain a rendering code.

[0136] According to an embodiment of the present disclosure, the image features of the target image are embedded in the text features of the initial guide text to obtain a rendering code, including: determining the image code of the target image and the text code of the initial guide text; and superimposing the image code and the text code to obtain a rendering code.

[0137] According to an embodiment of the present disclosure, since the target image and the initial guide text have different data formats, it is difficult to directly superimpose or embed them. Therefore, the target image and the initial guide text can be converted into feature vectors respectively, and then the image encoding of the target image and the text encoding of the initial guide text are superimposed to obtain a rendering code for further rendering the target image.

[0138] According to the embodiments of the present disclosure, by fusing and embedding the initial guidance text determined based on the target attribute information with the image code of the target image, the text features of the target attribute information and the image features of the target image can be integrated to obtain a rendering code used as a prompt word. Using the rendering code to render the target image further enhances the details and texture of the target image, thereby optimizing the image details and making the generated image more realistic and accurate.

[0139] Figure 8 The structure diagram of image rendering of the image generation method according to an embodiment of the present disclosure is schematically shown.

[0140] like Figure 8 As shown in the figure, random noise is added to the image. Using a UNet, the rendered image is used to predict the noise and perform reverse denoising. During training, a variational autoencoder can be used to extract the latent image information. In application, the variational autoencoder can be used to decode the latent image information into an image, resulting in the rendered target image.

[0141] Among them, in the second stage of fine-tuning the model according to the rendering loss during the model fine-tuning process, the fine-tuned model parameters may include the model parameters of UNet.

[0142] Based on the above model training method, the present disclosure also provides a model training device. Figure 9 The device is described in detail.

[0143] Figure 9 The structural block diagram of the model training device according to an embodiment of the present disclosure is schematically shown.

[0144] like Figure 9 As shown, the model training device 900 of this embodiment includes a first image transformation module 910, a second image transformation module 920, a first loss determination module 930, a second loss determination module 940, a third loss determination module 950 and a first model training module 960.

[0145] The first image transformation module 910 is configured to perform an image transformation on the sample object in the original sample image using the first generator based on the target sample attribute information to generate a target sample image, wherein the target sample attribute information is different from the original attribute information of the sample object in the original sample image. In one embodiment, the first image transformation module 910 can be configured to perform operation S210 described above and will not be further described herein.

[0146] The second image transformation module 920 is used to use the second generator to perform image transformation on the sample object in the target sample image based on the original attribute information to generate a restored sample image. In one embodiment, the second image transformation module 920 can be used to perform the operation S220 described above, which will not be repeated here.

[0147] The first loss determination module 930 is configured to determine a first loss value based on feature similarity between the original sample image and the target sample image. In one embodiment, the first loss determination module 930 may be configured to perform the aforementioned operation S230, which will not be described in detail herein.

[0148] The second loss determination module 940 is configured to use a first discriminator matched with the first generator to perform similarity discrimination on the original sample image and the target sample image to determine a second loss value. In one embodiment, the second loss determination module 940 can be configured to perform operation S240 described above and will not be further described here.

[0149] The third loss determination module 950 is used to determine a third loss value based on the feature similarity between the original sample image and the restored sample image. In one embodiment, the third loss determination module 950 can be used to perform the operation S250 described above, which will not be repeated here.

[0150] The first model training module 960 is used to train the first generator based on the first loss value, the second loss value, and the third loss value to obtain the trained first generator. In one embodiment, the first model training module 960 can be used to perform the operation S260 described above, which will not be repeated here.

[0151] According to an embodiment of the present disclosure, the model training device 900 further includes a fourth loss determination module, a fifth loss determination module, a sixth loss determination module and a second model training module.

[0152] The fourth loss determination module is used to determine a fourth loss value based on feature similarity between the target sample image and the restored sample image.

[0153] The fifth loss determination module is used to use the second discriminator matched with the second generator to perform similarity judgment on the target sample image and the restored sample image to determine the fifth loss value.

[0154] The sixth loss determination module is used to determine a sixth loss value based on feature similarity between the original sample image and the restored sample image.

[0155] The second model training module is used to train the second generator based on the fourth loss value, the fifth loss value and the sixth loss value to obtain the trained second generator.

[0156] According to an embodiment of the present disclosure, the first model training module 960 includes a parameter exchange submodule and a retraining submodule.

[0157] The parameter exchange submodule is used to exchange the model parameters of the trained second generator and the trained first generator to obtain a new first generator and a new second generator when it is determined that the trained first generator does not meet the training conditions.

[0158] The retraining submodule is used to use the new first generator and the new second generator to retrain based on the original image, target sample attribute information and original attribute information to obtain a new trained first generator.

[0159] According to an embodiment of the present disclosure, the first image transformation module 910 includes a parameter determination submodule and a feature transformation submodule.

[0160] The parameter determination submodule is used to determine the feature change parameter based on the first average object feature corresponding to the original sample attribute information and the second average object feature corresponding to the target sample attribute information, wherein the first average object feature is a feature point group extracted from the original sample, and the second average object feature is a feature point group generated by using the first generator.

[0161] The feature transformation submodule is used to transform the object features of the sample objects in the original sample image according to the feature change parameters to generate a target sample image.

[0162] According to an embodiment of the present disclosure, the model training device 900 further includes: a reference feature determination module and an average feature determination module.

[0163] The reference feature determination module is used to determine the reference features of the objects included in each of the multiple target reference images based on the multiple target reference images that match the preset attribute information.

[0164] The average feature determination module is used to determine the average object feature of the preset attribute information based on the average value of multiple reference features.

[0165] According to an embodiment of the present disclosure, the parameter determination submodule includes a deviation determination unit and a parameter determination unit.

[0166] The deviation determining unit is configured to determine, for each pair of corresponding first average feature points and second average feature points, a position deviation between the first average feature point and the second average feature point.

[0167] The parameter determination unit is used to determine the feature change parameter between the first average feature point and the second average feature point according to the position deviation.

[0168] According to an embodiment of the present disclosure, the feature transformation submodule includes a feature translation unit.

[0169] The feature translation unit is used to perform translation processing on the object features of the sample object according to the feature change direction and feature change amplitude to generate a target sample image.

[0170] Figure 10 The structural block diagram of the image generating device according to an embodiment of the present disclosure is schematically shown.

[0171] like Figure 10 As shown, the model training device 1000 of this embodiment includes an image generation module 1010.

[0172] Image generation module 1010 is configured to generate a target image based on the target attribute information and the original attribute information of the target object in the original image, using a target generator, where the target generator is trained using the aforementioned model training device 900. In one embodiment, image generation module 1010 can be configured to perform operation S710 described above, and will not be further described here.

[0173] According to an embodiment of the present disclosure, the model training device 1000 further includes a text generation module, a feature embedding module and an image rendering module.

[0174] The text generation module is used to generate initial guidance text based on target attribute information.

[0175] The feature embedding module is used to embed the image features of the target image into the text features of the initial guide text to obtain a rendering code.

[0176] The image rendering module is used to render the target image according to the rendering code to obtain the rendered target image.

[0177] According to an embodiment of the present disclosure, the feature embedding module includes a coding determination submodule, a coding superposition submodule and a decoding submodule.

[0178] The encoding determination submodule is used to determine the image encoding of the target image and the text encoding of the initial guide text.

[0179] The code superposition submodule is used to superimpose the image code and the text code to obtain the rendering code.

[0180] According to an embodiment of the present disclosure, any multiple modules among the first image transformation module 910, the second image transformation module 920, the first loss determination module 930, the second loss determination module 940, the third loss determination module 950, the first model training module 960, and the image generation module 1010 can be combined into a single module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to an embodiment of the present disclosure, at least one of the first image transformation module 910, the second image transformation module 920, the first loss determination module 930, the second loss determination module 940, the third loss determination module 950, the first model training module 960, and the image generation module 1010 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware by any other reasonable means of circuit integration or packaging, or implemented in any one of the three implementation modes of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the first image transformation module 910, the second image transformation module 920, the first loss determination module 930, the second loss determination module 940, the third loss determination module 950, the first model training module 960, and the image generation module 1010 may be at least partially implemented as a computer program module, which, when executed, may perform the corresponding function.

[0181] Figure 11 A block diagram of an electronic device suitable for implementing a model training method and an image generation method according to an embodiment of the present disclosure is schematically shown.

[0182] like Figure 11 As shown, the electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage portion 1108 into a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0183] Various programs and data required for the operation of the electronic device 1100 are stored in the RAM 1103. The processor 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. The processor 1101 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1102 and / or the RAM 1103. It should be noted that the programs may also be stored in one or more memories other than the ROM 1102 and the RAM 1103. The processor 1101 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0184] According to an embodiment of the present disclosure, electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to bus 1104. Electronic device 1100 may also include one or more of the following components connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 1108 including a hard disk; and a communication section 1109 including a network interface card such as a LAN card or modem. Communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1110 as needed, so that computer programs read from the removable media can be installed into storage section 1108 as needed.

[0185] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0186] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1102 and / or RAM 1103 described above, and / or one or more memories other than ROM 1102 and RAM 1103.

[0187] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiments of the present disclosure.

[0188] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the computer program is executed by the processor 1101. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0189] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1109, and / or installed from removable media 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0190] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1109 and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0191] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0193] Those skilled in the art will appreciate that the features described in the various embodiments of the present disclosure may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of the present disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of the present disclosure. All such combinations and / or couplings fall within the scope of the present disclosure.

[0194] The above describes the embodiments of the present disclosure. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A model training method, characterized in that: The method comprises: Based on the target sample attribute information, using a first generator to perform image transformation on the sample object in the original sample image to generate a target sample image, wherein the target sample attribute information is different from the original attribute information of the sample object in the original sample image; Using a second generator, performing image transformation on the sample object in the target sample image based on the original attribute information to generate a restored sample image; Determining a first loss value based on feature similarity between the original sample image and the target sample image; Using a first discriminator matched with the first generator to perform similarity discrimination on the original sample image and the target sample image, and determine a second loss value; determining a third loss value based on feature similarity between the original sample image and the restored sample image; Based on the first loss value, the second loss value, and the third loss value, the first generator is trained to obtain a trained first generator.

2. The method according to claim 1, characterized in that The method further comprises: determining a fourth loss value based on feature similarity between the target sample image and the restored sample image; Using a second discriminator matched with the second generator to perform similarity discrimination on the target sample image and the restored sample image to determine a fifth loss value; determining a sixth loss value based on feature similarity between the original sample image and the restored sample image; Based on the fourth loss value, the fifth loss value and the sixth loss value, the second generator is trained to obtain a trained second generator.

3. The method according to claim 2, characterized in that The step of training the first generator based on the first loss value, the second loss value, and the third loss value includes: When it is determined that the trained first generator does not meet the training condition, the model parameters of the trained second generator and the trained first generator are exchanged to obtain a new first generator and a new second generator; The new first generator and the new second generator are used to retrain based on the original image, the target sample attribute information and the original attribute information to obtain a new trained first generator.

4. The method according to claim 1, wherein The method of performing image transformation on the sample object in the original sample image by using the first generator based on the target sample attribute information to generate the target sample image includes: The following operations are performed using the first generator: determining a feature variation parameter based on a first average object feature corresponding to the original sample attribute information and a second average object feature corresponding to the target sample attribute information, wherein the first average object feature is determined using the object features of each of the plurality of objects corresponding to the original sample attribute information, and the second average object feature is determined using the object features of each of the plurality of objects corresponding to the target sample attribute information; and The object features of the sample object in the original sample image are transformed according to the feature change parameters to generate the target sample image.

5. The method according to claim 4, characterized in that The average object feature includes the first average object feature or the second average object feature; The average object feature is determined as follows: Determining, based on a plurality of target reference images that match preset attribute information, reference features of objects included in each of the plurality of target reference images; and An average object feature of the preset attribute information is determined according to an average value of the plurality of reference features.

6. The method according to claim 5, characterized in that The first average object feature includes a plurality of first average feature points, and the second average object feature includes a plurality of second average feature points, wherein the first average feature points and the second average feature points correspond to each other one by one; The determining of the feature change parameter according to the first average object feature corresponding to the original sample attribute information and the second average object feature corresponding to the target sample attribute information includes: For each pair of corresponding first average feature points and second average feature points, determining a positional deviation between the first average feature point and the second average feature point; and A feature variation parameter between the first average feature point and the second average feature point is determined according to the position deviation.

7. The method according to claim 6, characterized in that The characteristic change parameters include characteristic change direction and characteristic change amplitude; The transforming the object features of the sample object in the original sample image according to the feature change parameter to generate the target sample image includes: The object features of the sample object are translated according to the feature change direction and the feature change amplitude to generate the target sample image.

8. An image generation method, characterized in that: The method comprises: Based on the target attribute information and the original attribute information of the target object in the original image, a target image is generated using a target generator, wherein the target generator is trained according to the method according to any one of claims 1 to 7.

9. The method according to claim 8, characterized in that The method further comprises: generating an initial guidance text based on the target attribute information; Embedding the image features of the target image into the text features of the initial guide text to obtain a rendering code; and The target image is rendered according to the rendering code to obtain a rendered target image.

10. The method according to claim 9, characterized in that The step of embedding the image features of the target image into the text features of the initial guide text to obtain a rendering code includes: determining an image code of the target image and a text code of the initial guide text; and The image code and the text code are superimposed to obtain the rendering code.

Images