Image processing model training method, image generation method, device and equipment

By determining and fusing the adjustable attributes and fixed attribute feature vectors of the target object in the image processing model and adjusting the model parameters, the limitations of the image processing model in the prior art on discrete states are solved, and the continuity of the target object attributes is achieved. Processing and specific attribute control improves the efficiency and accuracy of image generation.

CN112785495BActive Publication Date: 2024-11-19UISEE TECH (ZHEJIANG) LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110113482.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-27
Publication Date
2024-11-19
Estimated Expiration
2041-01-27

AI Technical Summary

Technical Problem

Existing image processing models can only process a limited number of discrete states of target object properties on the image, and cannot effectively separate the adjustable and fixed attributes of the target object, resulting in the inability to implement control and image generation for specific attribute states.

Method used

By determining the adjustable attributes and fixed attribute feature vectors of the target object, the vector fusion is performed and the input is generated to generate an adversarial network, the target image is generated, and the model parameters are adjusted based on the degree of difference until the image generation needs are met.

Benefits of technology

Continuity processing of target object attributes on the image is realized, state control and image generation can be performed for specific attributes, and the efficiency and accuracy of image generation are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112785495B_ABST
    Figure CN112785495B_ABST
Patent Text Reader

Abstract

The disclosed embodiments relate to an image processing model training method, an image generation method, an apparatus and a device, the training method comprising: determining a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on a sample image; performing vector fusion on the first adjustable attribute feature vector and the first fixed attribute feature vector, and inputting the vectors into a generator in a generative adversarial network to generate a target image; determining a second adjustable attribute feature vector and a second fixed attribute feature vector of the target object on the target image; determining the adjustable attribute difference based on the first adjustable attribute feature vector and the second adjustable attribute feature vector; determining the fixed attribute difference based on the first fixed attribute feature vector and the second fixed attribute feature vector; adjusting the network parameters of the image processing model based on the adjustable attribute difference and the fixed attribute difference. The disclosed embodiments achieve continuous processing of target object attributes and state control of specific attributes of the target object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing model training method, image generation method, device and equipment. Background Art

[0002] Currently, image generation technology is widely studied as an effective way to synthesize new images. Among them, manipulating specific properties of object instances in images is not only conducive to more controllable image editing, but also to many image understanding tasks, such as state estimation and recognition of object instances in images in fields such as unmanned or autonomous driving.

[0003] Manipulating specific attributes of object instances on images can be viewed as a transformation of attributes from one state to another, and considering states as domains / types, attribute manipulation can be modeled as domain / type transformation. In existing attribute transformation schemes based on model training, on the one hand, due to the limitations of model volume and the number of training samples under each attribute state, such as using sample images of object instances in some specific attribute states for model training, the trained model can only process a limited number of discrete states of object instance attributes; on the other hand, since the model can only adapt to the discrete states of object instance attributes, the attributes to be adjusted of the object instance and other attribute information cannot be well separated during the model processing process, and thus the state control and image generation for a specific attribute cannot be achieved. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide an image processing model training method, an image generation method, an apparatus and a device.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing model training method, wherein the image processing model includes a generative adversarial network, and the method includes:

[0006] Determine a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on a sample image; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object;

[0007] After vector fusion of the first adjustable attribute feature vector and the first fixed attribute feature vector, the vectors are input into a generator in the generative adversarial network to generate a target image corresponding to the sample image;

[0008] Determine a second adjustable attribute feature vector and a second fixed attribute feature vector of the target object on the target image;

[0009] Determining an adjustable attribute difference based on the first adjustable attribute feature vector and the second adjustable attribute feature vector;

[0010] Determining a fixed attribute difference based on the first fixed attribute feature vector and the second fixed attribute feature vector;

[0011] Based on the adjustable attribute difference and the fixed attribute difference, a network parameter of the image processing model is adjusted.

[0012] In a second aspect, an embodiment of the present disclosure further provides an image generation method, which is implemented based on a pre-trained image processing model, and the image processing model is obtained based on any of the image processing model training methods provided in the embodiments of the present disclosure, and the image processing model includes a generative adversarial network, and the image generation method includes:

[0013] Determine a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on the image to be processed; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object;

[0014] After vector fusion of the first adjustable attribute feature vector and the first fixed attribute feature vector, the vectors are input into a generator in the generative adversarial network to generate a target image corresponding to the image to be processed.

[0015] In a third aspect, an embodiment of the present disclosure further provides an image processing model training device, wherein the image processing model includes a generative adversarial network, and the device includes:

[0016] A first feature vector determination module, used to determine a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on a sample image; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object;

[0017] a target image generation module, configured to perform vector fusion on the first adjustable attribute feature vector and the first fixed attribute feature vector, and input the resultant vector into a generator in the generative adversarial network to generate a target image corresponding to the sample image;

[0018] A second feature vector determination module, used to determine a second adjustable attribute feature vector and a second fixed attribute feature vector of the target object on the target image;

[0019] An adjustable attribute difference determination module, configured to determine an adjustable attribute difference based on the first adjustable attribute feature vector and the second adjustable attribute feature vector;

[0020] A fixed attribute difference determination module, configured to determine a fixed attribute difference based on the first fixed attribute feature vector and the second fixed attribute feature vector;

[0021] A network parameter adjustment module is used to adjust the network parameters of the image processing model based on the adjustable attribute difference and the fixed attribute difference.

[0022] In a fourth aspect, an embodiment of the present disclosure further provides an image generation device, wherein the image generation device is implemented based on a pre-trained image processing model, wherein the image processing model is obtained based on any of the image processing model training methods provided in the embodiments of the present disclosure, wherein the image processing model includes a generative adversarial network, and wherein the image generation device includes:

[0023] A feature vector determination module, used to determine a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on the image to be processed; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object;

[0024] The target image generation module is used to perform vector fusion on the first adjustable attribute feature vector and the first fixed attribute feature vector, and input the result into the generator in the generative adversarial network to generate a target image corresponding to the image to be processed.

[0025] In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the electronic device implements any image processing model training method or image generation method provided in the embodiments of the present disclosure.

[0026] In a sixth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a computing device, the computing device implements any image processing model training method or image generation method provided by an embodiment of the present disclosure.

[0027] Compared with the prior art, the technical solution provided by the disclosed embodiment has at least the following advantages: in the disclosed embodiment, by separating the adjustable attributes and the fixed attributes of the target object based on the attribute adjustment requirements of the target object on the sample image, and determining the first adjustable attribute feature vector of the target object based on the attribute adjustment requirements; then combining the first fixed attribute feature vector of the target object, the vectors are fused and input into the generator of the generative adversarial network in the image processing model to obtain a target image corresponding to the sample image; finally, based on the obtained adjustable attribute feature vectors and fixed attribute feature vectors, the adjustable attribute difference and the fixed attribute difference are obtained, and the network parameters of the image processing model are adjusted. The aforementioned operation can be repeated to perform multiple cycles of network parameter adjustment until an image processing model that meets the image generation requirements is obtained. The embodiments of the present disclosure solve the problem that existing models in the field of image processing can only process a limited number of discrete states of the attributes of the target object on the image, and the problem that the adjustable attributes and fixed attributes of the target object cannot be effectively separated during the model processing process, and thus the state control and image generation for a certain specific attribute of the target object cannot be realized. The image processing model trained by using the technical solution of the embodiments of the present disclosure can realize continuous processing of the attributes of the target object on the image, without being limited to processing discrete states of the attributes of the target object, and can realize state control and image generation for a certain specific attribute of the target object. For example, by controlling the state of a certain specific attribute of the target object, an image that meets the attribute adjustment requirements of the target object can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0030] Figure 1 A flowchart of an image processing model training method provided in an embodiment of the present disclosure;

[0031] Figure 2 A schematic diagram of a training architecture of an image processing model provided in an embodiment of the present disclosure;

[0032] Figure 3 A flowchart of another image processing model training method provided in an embodiment of the present disclosure;

[0033] Figure 4A schematic diagram of a training architecture of another image processing model provided in an embodiment of the present disclosure;

[0034] Figure 5 A flowchart of another image processing model training method provided in an embodiment of the present disclosure;

[0035] Figure 6 A schematic diagram of a training architecture of another image processing model provided in an embodiment of the present disclosure;

[0036] Figure 7 A flowchart of another image processing model training method provided in an embodiment of the present disclosure;

[0037] Figure 8 A schematic diagram of a training architecture of another image processing model provided in an embodiment of the present disclosure;

[0038] Fig. 9 A flowchart of an image generation method provided by an embodiment of the present disclosure;

[0039] Fig.10 A schematic diagram of the structure of an image processing model training device provided in an embodiment of the present disclosure;

[0040] Fig.11 A schematic diagram of the structure of an image generating device provided by an embodiment of the present disclosure;

[0041] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0043] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0044] Figure 1A flowchart of an image processing model training method provided for an embodiment of the present disclosure can be applied to the situation of how to perform model training based on the attribute adjustment requirements of the target object on the image. The image processing model training method can be executed by an image processing model training device, which can be implemented in software and / or hardware and can be integrated on any electronic device with computing capabilities, such as a terminal or a server. Furthermore, the electronic device can also be integrated on other devices with image processing requirements, such as autonomous driving vehicles, unmanned vehicles or drones, and can be deployed specifically according to actual conditions.

[0045] It should be understood that Figure 1 There is no strict order of execution between the operations shown in Figure 1 The execution order shown is understood as a specific limitation of the embodiment of the present disclosure. The execution order between various operations can be adjusted according to actual processing conditions. Different operations can be executed in parallel or in series.

[0046] Figure 2 A schematic diagram of a training architecture of an image processing model provided in an embodiment of the present disclosure, for combining Figure 1 The embodiments of the present disclosure are exemplarily described, but should not be understood as specific limitations of the embodiments of the present disclosure. Figure 2 As shown, the image processing model may include a Generative Adversarial Network (GAN), which includes a generator and a discriminator. The technical principle of the Generative Adversarial Network itself can refer to the prior art, and the embodiments of the present disclosure are not specifically limited.

[0047] like Figure 1 As shown, the image processing model training method provided by the embodiment of the present disclosure may include:

[0048] S101, determining a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on a sample image; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object.

[0049] During the model training process, the sample images may be any type of images, such as human images, building images, landscape images, etc., and the target objects may include the target photographed objects on the sample images, such as people on human images, buildings on building images, and scenery on landscape images. The adjustable attributes of the target objects on the sample images refer to the attributes that need to be controlled (or adjusted), and the fixed attributes (or invariant attributes) refer to the attributes that need to keep the state unchanged.

[0050] like Figure 2As shown, taking the sample image as a person image as an example, the target object refers to the person in the sample image, the facial orientation of the person can be used as an adjustable attribute, and other attributes of the person except the facial orientation can be used as fixed attributes, such as skin color, hairstyle, clothing color, facial expression, and background, etc., and any target image with a changed facial orientation and unchanged fixed attributes can be generated by controlling the continuous state value of the facial orientation (different state values ​​correspond to different adjustable attribute feature vectors).

[0051] In some embodiments, after determining the adjustable attributes and fixed attributes of the target object on the sample image, any available image feature sampling (or feature extraction) method can be used to determine the first fixed attribute feature vector of the target object on the sample image based on the fixed attributes of the target object, and generate the first adjustable attribute feature vector corresponding to the adjustable attribute according to the adjustment requirements of the adjustable attribute (for example, the face of the person is rotated 15 degrees to the left). For example, the corresponding relationship between the state value and the vector modulus value of the adjustable attribute of the target object in the attribute space can be determined in advance, and then the required state value is determined according to the attribute adjustment requirements of the target object, and then the vector modulus value corresponding to the state value is determined, and finally, according to the determined vector modulus value, the first adjustable attribute feature vector is generated based on any available feature vector generation method.

[0052] Continue as Figure 2 As shown, taking a face image as an example, assuming that the face orientation is an adjustable attribute of the target object, the correspondence between the state value [0°, 90°] of the face orientation in the attribute space and the vector modulus value [0, 0.5] is predetermined, and the initial state value of the face orientation on the face image is 0° (i.e., in a frontal face state). If the state value of the current requirement is determined to be 90° (e.g., turned 90° toward its left shoulder, in a side face state) according to the face orientation adjustment requirement of the target object, the corresponding vector modulus value is 0.5, and then a first adjustable attribute feature vector is generated based on the vector modulus value. It should be understood that when the adjustable attribute state takes any other value, the corresponding vector modulus value and the first adjustable attribute feature vector can also be determined, that is, the disclosed embodiment can achieve continuity control of the adjustable attribute state without being limited to a specific discrete attribute state.

[0053] S102: After vector fusion, the first adjustable attribute feature vector and the first fixed attribute feature vector are input into a generator in a generative adversarial network to generate a target image corresponding to the sample image.

[0054] like Figure 2As shown, after the first adjustable attribute feature vector and the first fixed attribute feature vector are fused (fused), they are input into the generator in the generative adversarial network to generate a target image. At the same time, the discriminator in the generative adversarial network can identify the target image generated by the generator based on the sample image, that is, the generative adversarial network can use the sample image as real image data to achieve a self-supervised image generation effect.

[0055] S103: Determine a second adjustable attribute feature vector and a second fixed attribute feature vector of the target object on the target image.

[0056] Exemplarily, any available image feature extraction method may be used to determine the second adjustable attribute feature vector and the second fixed attribute feature vector of the target object on the target image, which is not specifically limited in the embodiments of the present disclosure.

[0057] S104: Determine the adjustable attribute difference based on the first adjustable attribute feature vector and the second adjustable attribute feature vector.

[0058] The adjustable attribute difference can be used to measure the difference between the ideal adjustable attribute state value of the target object (i.e., the adjustable attribute state value predetermined based on the attribute adjustment requirements of the target object) and the actual adjustable attribute state value of the target object on the target image, i.e., the difference between the required effect and the actual generated effect. Any algorithm that can be used to determine the difference between two vectors can be used to calculate the adjustable attribute difference.

[0059] In some embodiments, based on the first adjustable attribute feature vector and the second adjustable attribute feature vector, the adjustable attribute difference is determined, including: determining the modulus value of the second adjustable attribute feature vector, and determining the difference or quotient between the modulus value and the modulus value corresponding to the first adjustable attribute feature vector, and using the difference or quotient as the adjustable attribute difference; wherein the modulus value corresponding to the first adjustable attribute feature vector is predetermined according to the attribute adjustment requirements of the target object, and the difference mentioned in the embodiments of the present disclosure may be a positive value.

[0060] The value of the vector modulus value is usually between 0 and 1. The specific calculation of the vector modulus value can be implemented by referring to the prior art, and is not specifically limited in the embodiments of the present disclosure. It should be noted that when determining the correspondence between the state value of the adjustable attribute of the target object in the attribute space and the vector modulus value, the calculation method of the vector modulus value is the same as the calculation method of determining the modulus value of the second adjustable attribute feature vector in the process of determining the adjustable attribute difference, thereby ensuring the calculation accuracy of the adjustable attribute difference.

[0061] S105 . Determine a fixed attribute difference based on the first fixed attribute feature vector and the second fixed attribute feature vector.

[0062] The fixed attribute difference can be used to measure whether the fixed attributes of the target object on the sample image and the target image have changed, that is, whether the fixed attributes of the target object have changed before and after the attribute adjustment. The fixed attribute difference can be calculated in the same way as the adjustable attribute difference.

[0063] Exemplarily, determining the fixed attribute difference based on the first fixed attribute feature vector and the second fixed attribute feature vector includes: respectively determining the modulus values ​​of the first fixed attribute feature vector and the second fixed attribute feature vector, and taking the difference or quotient between the two modulus values ​​as the fixed attribute difference.

[0064] S106. Adjust the network parameters of the image processing model based on the adjustable attribute difference and the fixed attribute difference.

[0065] During the model training process, the difference thresholds corresponding to the adjustable attribute difference and the fixed attribute difference can be pre-set according to the model training accuracy (the value can be flexibly set). If the adjustable attribute difference and the fixed attribute difference are both less than the corresponding difference thresholds, it means that the difference between the ideal adjustable attribute state value of the target object and the actual adjustable attribute state value of the target object on the target image is within the allowable error range, and the fixed attributes of the target object on the sample image and the target image have not changed, the image accuracy currently output by the model meets the requirements, and the model training can be terminated; if the adjustable attribute difference and the fixed attribute difference are both greater than or equal to the corresponding difference thresholds, it means that there is a large difference between the ideal adjustable attribute state value of the target object and the actual adjustable attribute state value of the target object on the target image, and the fixed attributes of the target object on the sample image and the target image have changed, and the image accuracy currently output by the model does not meet the requirements, and it is necessary to adjust the network parameters of the image processing model according to the adjustable attribute difference and the fixed attribute difference, and optimize the model until the adjustable attribute difference and the fixed attribute difference obtained by repeatedly executing the model training operation are respectively less than the corresponding difference thresholds. Of course, if the adjustable attribute difference or the fixed attribute difference is greater than or equal to the corresponding difference threshold, it is also necessary to adjust the network parameters of the image processing model according to the adjustable attribute difference and the fixed attribute difference to optimize the model. The network parameters can be determined according to the specific model structure, and those skilled in the art can reasonably configure the network parameters, which are not specifically limited in the embodiments of the present disclosure.

[0066] In the disclosed embodiment, the adjustable attributes and fixed attributes of the target object are separated based on the attribute adjustment requirements of the target object on the sample image, and the first adjustable attribute feature vector of the target object is determined based on the attribute adjustment requirements; then, combined with the first fixed attribute feature vector of the target object, it is input into the generator of the generative adversarial network in the image processing model through vector fusion to obtain a target image corresponding to the sample image; finally, based on the obtained adjustable attribute feature vectors and fixed attribute feature vectors, the adjustable attribute difference and the fixed attribute difference are obtained, and the network parameters of the image processing model are adjusted. The aforementioned operation can be repeated to perform multiple cycles of network parameter adjustment until an image processing model that meets the image generation requirements is obtained. The embodiments of the present disclosure solve the problem that existing models in the field of image processing can only process a limited number of discrete states of the attributes of the target object on the image, and the problem that the adjustable attributes and fixed attributes of the target object cannot be effectively separated during the model processing process, and thus the state control and image generation for a certain specific attribute of the target object cannot be realized. The image processing model trained by using the technical solution of the embodiments of the present disclosure can realize continuous processing of the attributes of the target object on the image, without being limited to processing discrete states of the attributes of the target object, and can realize state control and image generation for a certain specific attribute of the target object. For example, by controlling the state of a certain specific attribute of the target object, an image that meets the attribute adjustment requirements of the target object can be generated.

[0067] Figure 3 The flowchart of another image processing model training method provided in the embodiment of the present disclosure is further optimized and expanded based on the above technical solution, and can be combined with the above optional implementations. It should be understood that Figure 3 There is no strict order of execution between the operations shown in Figure 3 The execution order shown is understood as a specific limitation of the embodiment of the present disclosure. The execution order between various operations can be adjusted according to actual processing conditions. Different operations can be executed in parallel or in series.

[0068] Figure 4 A schematic diagram of a training architecture of another image processing model provided in an embodiment of the present disclosure, taking a sample image as a person image as an example, for combining Figure 3 The embodiments of the present disclosure are exemplarily described, but should not be understood as specific limitations of the embodiments of the present disclosure. Figure 4As shown, the image processing model may include a generative adversarial network, a capsule network and a multi-branch estimator, wherein the generative adversarial network includes a generator and a discriminator, and the multi-branch estimator includes a convolutional layer and a plurality of branch capsule networks respectively connected to the convolutional layer, and the number of convolutional layers may be determined according to demand, and the embodiment of the present disclosure does not specifically limit it. Both the capsule network and the multi-branch estimator can be used to extract image features.

[0069] It should be noted that Figure 4 In order to show the processing flow of image data, the image processing model including the capsule network is repeatedly drawn on the right side of the target image. This does not mean that there are two image processing models in the disclosed embodiment. It is intended to clearly show that the second fixed feature vector and the first fixed feature vector of the target object are both input into the capsule network for processing. In the image processing model, the number of capsule networks can be one or more according to the processing requirements of the feature vector. For example, the image processing model can include two capsule networks, which are used to process the second fixed feature vector and the first fixed feature vector respectively. The parameters of multiple capsule networks can be the same or different, which can be determined according to the training requirements of the model.

[0070] like Figure 3 As shown, the image processing model training method provided by the embodiment of the present disclosure may include:

[0071] S301, determining a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on a sample image; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object.

[0072] S302: After vector fusion, the first adjustable attribute feature vector and the first fixed attribute feature vector are input into a generator in a generative adversarial network to generate a target image corresponding to the sample image.

[0073] S303, inputting the target image and the image label corresponding to the target image into a multi-branch estimator to obtain a plurality of second adjustable attribute feature vectors of the target object on the target image; wherein the image label is used to mark the adjustable attribute.

[0074] The multi-branch estimator includes a convolutional layer and a plurality of branch capsule networks respectively connected to the convolutional layer, wherein each branch can extract features of the target image based on the image label corresponding to the target image to obtain a plurality of second adjustable attribute feature vectors of the target object on the target image. By using multi-branch estimation to extract features of the adjustable attributes of the target object on the target image, it is possible to focus on different aspects of the attribute state on the target image, and fuse the feature extraction results of each branch to obtain the adjustable attribute difference, which helps to improve the determination accuracy of the adjustable attribute difference.

[0075] S304: Determine a second fixed attribute feature vector of the target object on the target image.

[0076] S305: Determine a vector difference value between the first adjustable attribute feature vector and each second adjustable attribute feature vector.

[0077] S306: Determine an adjustable attribute difference based on the multiple vector difference values.

[0078] In some embodiments, the difference or quotient between the modulus value of the first adjustable attribute feature vector and the modulus value of each second adjustable attribute feature vector may be determined as the vector difference value between the first adjustable attribute feature vector and each second adjustable attribute feature vector; then the obtained multiple vector difference values ​​are averaged and the obtained average value is used as the final adjustable attribute difference degree, or the obtained multiple vector difference values ​​are summed and the obtained vector difference sum is used as the final adjustable attribute difference degree, etc. It should be understood that other methods capable of determining the adjustable attribute difference degree based on multiple vector difference values ​​may also be adopted.

[0079] Taking the difference between the modulus value of the first adjustable attribute feature vector and the modulus value of each second adjustable attribute feature vector (taking a positive value) as the vector difference value, and summing up multiple vector difference values, and taking the obtained vector difference sum as the final adjustable attribute difference as an example, that is, in the embodiment of the present disclosure, the adjustable attribute detection loss function related to the multi-branch estimator is It can be expressed as follows:

[0080]

[0081] Among them, ||·||1 represents the 1-norm, φ d Represents the modulus value of the second adjustable attribute feature vector of the estimator output of the d-th branch in the multi-branch estimator, α t Indicates the modulus value corresponding to the first adjustable attribute feature vector, I t Represents the target image.

[0082] The modulus value corresponding to the first adjustable attribute feature vector is predetermined according to the attribute adjustment requirements of the target object. The modulus value of each second adjustable attribute feature vector can also be directly output by the multi-branch estimator. The specific calculation of the modulus value can be implemented with reference to the prior art, and the embodiments of the present disclosure are not specifically limited. For example, for the capsule network, the following calculation method can be used to compress the modulus value of the capsule vector to less than 1:

[0083]

[0084] Among them, v j The vector representation after the modulus value of the capsule vector is compressed to less than 1, sj represents any capsule vector (or capsule vector), Used to maintain the original vector direction, is a scalar less than 1.

[0085] S307 , respectively inputting the first fixed attribute feature vector and the second fixed attribute feature vector into the capsule network, and determining the fixed attribute difference according to the output result of the capsule network.

[0086] The capsule network itself has a self-awareness mechanism that can support more powerful feature representation, and the vector representation of the capsule network can model the multi-peak distribution of image information to regress complex continuous signals. Therefore, the use of the capsule network can extract more accurate image features. Figure 4 As shown in , the first fixed attribute feature vector and the second fixed attribute feature vector are respectively input into the capsule network and converted into high-dimensional feature vectors, which helps to improve the calculation accuracy of the fixed attribute difference.

[0087] Among them, the output result of the capsule network can be a high-dimensional capsule network corresponding to the first fixed attribute feature vector and the second fixed attribute feature vector, or it can be the modulus value of the high-dimensional capsule network corresponding to the first fixed attribute feature vector and the second fixed attribute feature vector. The embodiment of the present disclosure is not specifically limited, and the difference or quotient between the modulus values ​​of the two high-dimensional capsule networks can be used as the fixed attribute difference.

[0088] Taking the difference between the modulus values ​​of two high-dimensional capsule networks (a positive value) as the fixed attribute difference as an example, that is, the fixed attribute detection loss function related to the capsule network in the embodiment of the present disclosure is It can be expressed as follows:

[0089]

[0090] Among them, σ(I s ) represents the sample image I s The capsule vector obtained after the first fixed attribute feature vector of the target object is input into the capsule network, σ(I t ) represents the target image I t The capsule vector obtained after the second fixed attribute feature vector of the target object is input into the capsule network, ||·||1 represents the 1-norm, and ||·||2 represents the 2-norm (corresponding to the modulus value).

[0091] S308. Adjust the network parameters of the image processing model based on the adjustable attribute difference and the fixed attribute difference.

[0092] Continue as Figure 4As shown, in some embodiments, the generative adversarial network in the image processing model further includes a multi-branch discriminator (that is, the discriminator in the generative adversarial network is implemented using a multi-branch discriminator), the number of branches of the multi-branch discriminator and the multi-branch estimator is the same, and the multi-branch discriminator and the multi-branch estimator share a convolutional layer ( Figure 4 The structural relationship is not shown in the figure); each branch of the multi-branch discriminator is used to discriminate the target image according to the sample image. Considering that there are often problems of gradient vanishing and mode collapse in the training process of the generative adversarial network, by using the multi-branch discriminator, the discriminator can focus on different image features, reducing the occurrence of gradient vanishing and mode collapse in the training process of the generative adversarial network.

[0093] In the disclosed embodiment, the first fixed attribute feature vector and the second fixed attribute feature vector of the target object are input into a capsule network, and the fixed attribute difference is determined according to the output result of the capsule network, thereby ensuring the calculation accuracy of the fixed attribute difference; a plurality of second adjustable attribute feature vectors of the target object on the target image are obtained by inputting the target image and the image label corresponding to the target image into a multi-branch estimator, and then the adjustable attribute difference is obtained by combining the first adjustable attribute feature vector, thereby ensuring the determination accuracy of the adjustable attribute difference; finally, based on the adjustable attribute difference and the fixed attribute difference, the network parameters of the image processing model are adjusted, so that the trained image processing model can adjust the requirements according to the attributes of the target object and generate an image that meets the requirements. That is, the embodiments of the present disclosure solve the problem that existing models in the field of image processing can only process a limited number of discrete states of the attributes of the target object on the image, and the problem that the adjustable attributes and fixed attributes of the target object cannot be effectively separated during the model processing process, and thus the state control and image generation for a certain specific attribute of the target object cannot be realized; the image processing model trained by using the technical solution of the embodiments of the present disclosure can realize the continuous processing of the attributes of the target object on the image, without being limited to processing the discrete states of the attributes of the target object, and can realize the state control and image generation for a certain specific attribute of the target object, thereby improving the generation efficiency of the required image.

[0094] Figure 5 The flowchart of another image processing model training method provided in the embodiment of the present disclosure is further optimized and expanded based on the above technical solution, and can be combined with the above optional implementations. It should be understood that Figure 5 There is no strict order of execution between the operations shown in Figure 5 The execution order shown is understood as a specific limitation of the embodiment of the present disclosure. The execution order between various operations can be adjusted according to actual processing conditions. Different operations can be executed in parallel or in series.

[0095] Figure 6 A schematic diagram of a training architecture of another image processing model provided in an embodiment of the present disclosure, taking a sample image as a person image as an example, for combining Figure 5 The embodiments of the present disclosure are exemplarily described, but should not be understood as specific limitations of the embodiments of the present disclosure. Figure 6 As shown, the image processing model may include a generative adversarial network, a capsule network, a multi-branch estimator, an encoder, and a decoder, wherein the generative adversarial network includes a generator and a multi-branch discriminator.

[0096] It should be noted that Figure 6 In order to show the processing flow of image data, the image processing model including the capsule network and the encoder is repeatedly drawn on the right side of the target image. This does not mean that there are two image processing models in the disclosed embodiment. It is intended to clearly show that the second fixed feature vector and the first fixed feature vector of the target object are both input into the capsule network for processing, and that the target image and the sample image can be extracted by the encoder. In the image processing model, according to the processing requirements of the feature vector, the number of capsule networks can be one or more. For example, the image processing model can include two capsule networks, which are used to process the second fixed feature vector and the first fixed feature vector respectively. Similarly, the number of encoders can also be one or more. For example, the image processing model can include two encoders, which are used to extract features from the sample image and the target image respectively. The parameters of multiple capsule networks or multiple encoders can be the same or different, which can be determined according to the training requirements of the model.

[0097] like Figure 5 As shown, the image processing model training method provided by the embodiment of the present disclosure may include:

[0098] S501: Input a sample image into an encoder to obtain a first fixed attribute feature vector of a target object.

[0099] The encoder has the function of extracting image features, and its specific implementation structure can be implemented with reference to the prior art, and the disclosed embodiment does not specifically limit it. Within the allowed range of image feature extraction error, the sample can be directly input into the encoder, and the output vector of the encoder is used as the first fixed attribute feature vector of the target object.

[0100] S502: Input a module length value predetermined according to an attribute adjustment requirement of a target object into a decoder to obtain a first adjustable attribute feature vector of the target object.

[0101] Among them, the decoder has the function of generating a feature vector according to the vector modulus value, and its specific implementation structure can be implemented with reference to the prior art, and the embodiments of the present disclosure are not specifically limited. Exemplarily, the attribute state value and the corresponding vector modulus value of the current attribute adjustment requirement can be determined according to the correspondence between the state value of the adjustable attribute of the target object in the attribute space and the vector modulus value, and then the vector modulus value is input into the decoder to obtain the first adjustable attribute feature vector of the target object.

[0102] S503: After vector fusion, the first adjustable attribute feature vector and the first fixed attribute feature vector are input into a generator in a generative adversarial network to generate a target image corresponding to the sample image.

[0103] S504, inputting the target image and the image label corresponding to the target image into a multi-branch estimator to obtain a plurality of second adjustable attribute feature vectors of the target object on the target image; wherein the image label is used to mark the adjustable attribute.

[0104] S505 : Input the target image into the encoder to obtain a second fixed attribute feature vector of the target object.

[0105] S506: Determine a vector difference value between the first adjustable attribute feature vector and each second adjustable attribute feature vector.

[0106] S507: Determine an adjustable attribute difference based on the multiple vector difference values.

[0107] S508 , respectively inputting the first fixed attribute feature vector and the second fixed attribute feature vector into the capsule network, and determining the fixed attribute difference according to the output result of the capsule network.

[0108] S509: Adjust the network parameters of the image processing model based on the adjustable attribute difference and the fixed attribute difference.

[0109] The technical solution of the embodiment of the present disclosure solves the problem that the existing models in the field of image processing can only process a limited number of discrete states of the attributes of the target object on the image, and the problem that the adjustable attributes and fixed attributes of the target object cannot be effectively separated during the model processing, and thus the state control and image generation for a certain specific attribute of the target object cannot be realized; the image processing model trained by the technical solution of the embodiment of the present disclosure can realize the continuous processing of the attributes of the target object on the image, without being limited to processing the discrete states of the attributes of the target object, and can realize the state control and image generation for a certain specific attribute of the target object, thereby improving the generation efficiency of the required image.

[0110] Figure 7The flowchart of another image processing model training method provided in the embodiment of the present disclosure is further optimized and expanded based on the above technical solution, and can be combined with the above optional implementations. It should be understood that Figure 7 There is no strict order of execution between the operations shown in Figure 7 The execution order shown is understood as a specific limitation of the embodiment of the present disclosure. The execution order between various operations can be adjusted according to actual processing conditions. Different operations can be executed in parallel or in series.

[0111] Figure 8 A schematic diagram of a training architecture of another image processing model provided in an embodiment of the present disclosure, taking a sample image as a person image as an example, for combining Figure 7 The embodiments of the present disclosure are exemplarily described, but should not be understood as specific limitations of the embodiments of the present disclosure. Figure 8 As shown, the image processing model may include a generative adversarial network, a capsule network, a multi-branch estimator, an encoder, and a decoder, wherein the generative adversarial network includes a generator and a multi-branch discriminator.

[0112] It should be noted that Figure 8 In order to show the processing flow of image data, the image processing model including the generative adversarial network, capsule network, multi-branch estimator, encoder and decoder is repeatedly drawn on the right side of the target image. This does not mean that there are two image processing models in the embodiment of the present disclosure. It is intended to clearly show the data flow. Figure 4 or Figure 6 Similarly, in the image processing model, according to the processing requirements of the feature vector or image, the number of generative adversarial networks, capsule networks, multi-branch estimators, encoders or decoders can be one or more, for example, the number of generative adversarial networks, capsule networks, multi-branch estimators, encoders or decoders in the image processing model is two. The parameters of the same functional modules in the image processing model can be the same or different, which can be determined according to the training requirements of the model.

[0113] like Figure 7 As shown, the image processing model training method provided by the embodiment of the present disclosure may include:

[0114] S701, determining a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on a sample image; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object.

[0115] S702: After vector fusion, the first adjustable attribute feature vector and the first fixed attribute feature vector are input into a generator in a generative adversarial network to generate a target image corresponding to the sample image.

[0116] S703: Determine a second adjustable attribute feature vector and a second fixed attribute feature vector of the target object on the target image.

[0117] S704: Determine the adjustable attribute difference based on the first adjustable attribute feature vector and the second adjustable attribute feature vector.

[0118] S705: Determine a fixed attribute difference based on the first fixed attribute feature vector and the second fixed attribute feature vector.

[0119] S706: Obtain an initial adjustable attribute feature vector of the target object on the sample image.

[0120] The initial adjustable attribute feature vector of the target object on the sample image refers to the feature vector corresponding to the adjustable attribute state value of the target object on the sample image before the target object attribute is adjusted. Figure 8 As shown, taking the sample image as a face image and the adjustable attribute as the face orientation as an example, before the face orientation is adjusted, the face orientation of the person is 0°, and the attribute state value corresponds to the initial adjustable attribute feature vector.

[0121] like Figure 8 As shown, in some embodiments, obtaining an initial adjustable attribute feature vector of a target object on a sample image includes:

[0122] Get the initial modulus value corresponding to the adjustable attribute of the target object;

[0123] The initial modulus value is input into the decoder in the image processing model to obtain an initial adjustable attribute feature vector.

[0124] Exemplarily, the initial attribute state value and the corresponding initial modulus value of the adjustable attribute of the target object on the sample image can be determined based on the correspondence between the state value of the adjustable attribute of the target object in the attribute space and the vector modulus value, and then the initial modulus value is input into the decoder to obtain the initial adjustable attribute feature vector.

[0125] Furthermore, the initial modulus value corresponding to the adjustable attribute of the target object is obtained, including:

[0126] A sample image and an image label corresponding to the sample image are input into a multi-branch estimator in an image processing model, and an initial modulus value corresponding to an adjustable attribute of a target object is determined according to an output result of the multi-branch estimator, wherein the image label is used to mark the adjustable attribute.

[0127] Exemplarily, the initial modulus value corresponding to the adjustable attribute of the target object output by the multi-branch estimator can be directly used, or the initial modulus value corresponding to the adjustable attribute of the target object can be calculated based on multiple initial vectors output by the multi-branch estimator. For example, the modulus values ​​of the multiple initial vectors output by the multi-branch estimator can be averaged, and the average value obtained can be used as the initial modulus value.

[0128] S707: After vector fusion, the initial adjustable attribute feature vector and the second fixed attribute feature vector are input into a generator in a generative adversarial network to generate a verification image corresponding to the target image.

[0129] That is, the verification image is an image generated again by restoring the state of the adjustable attributes of the target object based on the generated target image, and can be used to verify the quality of the generated target image and whether the target image meets the attribute adjustment requirements of the target object.

[0130] S708: Determine a third adjustable attribute feature vector of the target object on the verification image.

[0131] like Figure 8 As shown, after the initial adjustable attribute feature vector and the second fixed attribute feature vector are fused, they are input into the generator in the generative adversarial network to generate a verification image. At the same time, the verification image generated by the generator can be identified based on the sample image. That is, the generative adversarial network can use the sample image as real image data to achieve a self-supervised image generation effect.

[0132] After obtaining the verification image, any available image feature extraction method can be used to determine the third adjustable attribute feature vector of the target object on the verification image. Exemplarily, the verification image and the image label corresponding to the verification image can be input into a multi-branch estimator to obtain multiple third adjustable attribute feature vectors of the target object on the verification image; wherein the image label is used to mark the adjustable attribute.

[0133] S709: Determine an adjustable attribute verification value based on the initial adjustable attribute feature vector and the third adjustable attribute feature vector.

[0134] The adjustable attribute verification value can be used to measure the difference between the adjustable attribute state value of the target object on the verification image and the initial adjustable attribute state value of the target object on the sample image, thereby evaluating the generation quality of the target image and the training effect of the image processing model. The smaller the adjustable attribute verification value, the smaller the difference between the adjustable attribute state value of the target object on the verification image and the initial adjustable attribute state value of the target object on the sample image. The target image can be successfully restored to the sample image state, which means that the generated target image has a better quality and the current image processing model has a high probability of achieving a better training effect. Otherwise, it means that the generated target image has a poor quality and the current image processing model still needs to continue training.

[0135] In some embodiments, the difference or quotient between the initial modulus value corresponding to the initial adjustable attribute feature vector and the modulus value of the third adjustable attribute feature vector can be used as the adjustable attribute verification value. Exemplarily, the vector difference value between the initial adjustable attribute feature vector and each third adjustable attribute feature vector can be determined, and the adjustable attribute verification value can be determined based on multiple vector difference values, for example, the adjustable attribute verification value can be obtained by summing or averaging.

[0136] S710: Adjust the network parameters of the image processing model based on the adjustable attribute difference, the fixed attribute difference and the adjustable attribute verification value.

[0137] Specifically, during the model training process, the network parameters of the image processing model can be adjusted based on the adjustable attribute difference, the fixed attribute difference and the adjustable attribute verification value to improve the model training accuracy. Through the network parameter adjustment, until the adjustable attribute difference, the fixed attribute difference and the adjustable attribute verification value are all less than their corresponding difference thresholds (the values ​​can be flexibly set), it is considered that the model has achieved the training effect and the model training can be terminated, otherwise the aforementioned model training operation will continue.

[0138] In the embodiments of the present disclosure, a target image can be generated based on the attribute adjustment requirements of a sample image and a target object, and a verification image can be generated based on the attribute state restoration requirements of the target image and the target object. In combination with the extracted image feature vector, an adjustable attribute difference, a fixed attribute difference and an adjustable attribute verification value can be obtained, which can be used together to adjust the network parameters of the image processing model, thereby improving the training accuracy of the model and improving the effect of the model in generating the required image.

[0139] Fig. 9A flowchart of an image generation method provided for an embodiment of the present disclosure may be applicable to the case of image generation. The image generation method may be executed by an image generation device, which may be implemented in software and / or hardware and may be integrated on any electronic device with computing capabilities, such as a terminal or a server. Furthermore, the electronic device may also be integrated on other devices with image processing requirements, such as autonomous vehicles, unmanned vehicles, or drones, and may be deployed specifically according to actual conditions.

[0140] The image generation method provided in the embodiments of the present disclosure is implemented based on a pre-trained image processing model. The image processing model is obtained based on any image processing model training method provided in the embodiments of the present disclosure. The image processing model includes a generative adversarial network. For content not described in detail in the following embodiments, please refer to the description in the above embodiments.

[0141] like Fig. 9 As shown, the image generation method provided by the embodiment of the present disclosure may include:

[0142] S901, determining a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on an image to be processed; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object.

[0143] The image to be processed may be any type of image, such as a person image, a building image, a landscape image, etc. The target object may include the target photographed object on the image to be processed, such as a person on a person image, a building on a building image, a scene on a landscape image, etc. The adjustable attribute of the target object on the image to be processed refers to an attribute whose state can be controlled (or adjusted) during the image generation process, and the fixed attribute refers to an attribute whose state remains unchanged during the image generation process. Taking a person on a person image as an example, the face orientation of the person can be used as an adjustable attribute, and other attributes of the person except the face orientation can be used as fixed attributes, such as skin color, hairstyle, clothing color, facial expression, and background, etc. By controlling the continuous state value of the face orientation, any target image with a changed face orientation and unchanged fixed attributes can be generated.

[0144] In some embodiments, after determining the adjustable attributes and fixed attributes of the target object on the image to be processed according to the attribute adjustment requirements of the target object, any available image feature sampling method can be used to determine the first fixed attribute feature vector of the target object on the image to be processed based on the fixed attributes of the target object, and generate the first adjustable attribute feature vector corresponding to the adjustable attribute according to the adjustment requirements of the adjustable attribute. For example, the corresponding relationship between the state value and the vector modulus value of the adjustable attribute of the target object in the attribute space can be determined in advance, and then the state value of the requirement is determined according to the attribute adjustment requirements of the target object, and then the vector modulus value corresponding to the state value is determined, and finally, according to the determined vector modulus value, the first adjustable attribute feature vector is generated based on any available feature vector generation method.

[0145] In some embodiments, if the image processing model also includes an encoder and a decoder, determining the first adjustable attribute feature vector and the first fixed attribute feature vector of the target object on the image to be processed may include: inputting the image to be processed into the encoder to obtain the first fixed attribute feature vector of the target object; inputting the modulus value predetermined according to the attribute adjustment requirements of the target object into the decoder to obtain the first adjustable attribute feature vector of the target object.

[0146] S902: After vector fusion, the first adjustable attribute feature vector and the first fixed attribute feature vector are input into a generator in a generative adversarial network to generate a target image corresponding to the image to be processed.

[0147] For explanations of the image processing model, reference may be made to the contents of the above embodiments. The disclosed embodiment, by utilizing the image processing model, can quickly generate a target image that meets the attribute adjustment requirements based on the first adjustable attribute feature vector and the first fixed attribute feature vector of the target object on the image to be processed, thereby achieving effective state control of the specific attributes of the target object on the image and improving the generation efficiency of the required image.

[0148] Fig.10 This is a schematic diagram of the structure of an image processing model training device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and can be integrated on any electronic device with computing capabilities, such as a terminal or server. Furthermore, the electronic device can also be integrated on other devices with image processing requirements, such as autonomous driving vehicles, unmanned vehicles or drones, and can be deployed specifically according to actual conditions.

[0149] In the disclosed embodiment, the image processing model may include a generative adversarial network, such as Fig.10As shown, the image processing model training device 1000 provided in the embodiment of the present disclosure may include a first feature vector determination module 1001, a target image generation module 1002, a second feature vector determination module 1003, an adjustable attribute difference determination module 1004, a fixed attribute difference determination module 1005 and a network parameter adjustment module 1006, wherein:

[0150] A first feature vector determination module 1001 is used to determine a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on a sample image; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object;

[0151] A target image generation module 1002 is used to perform vector fusion on the first adjustable attribute feature vector and the first fixed attribute feature vector, and then input the result into a generator in a generative adversarial network to generate a target image corresponding to the sample image;

[0152] A second feature vector determination module 1003, used to determine a second adjustable attribute feature vector and a second fixed attribute feature vector of a target object on a target image;

[0153] An adjustable attribute difference determination module 1004 is used to determine the adjustable attribute difference based on the first adjustable attribute feature vector and the second adjustable attribute feature vector;

[0154] A fixed attribute difference determination module 1005, configured to determine a fixed attribute difference based on the first fixed attribute feature vector and the second fixed attribute feature vector;

[0155] The network parameter adjustment module 1006 is used to adjust the network parameters of the image processing model based on the adjustable attribute difference and the fixed attribute difference.

[0156] In some embodiments, the image processing model may further include a capsule network; accordingly, the fixed attribute difference determination module 1005 is specifically used for:

[0157] The first fixed attribute feature vector and the second fixed attribute feature vector are respectively input into the capsule network, and the fixed attribute difference is determined according to the output result of the capsule network. That is, the fixed attribute difference determination module 1005 can correspond to the capsule network in the image processing model, the capsule network has the function of extracting high-dimensional features from the sample image or the target image, and the output result of the capsule network can be used to determine the fixed attribute difference.

[0158] In some embodiments, the image processing model may further include a multi-branch estimator, the multi-branch estimator including a convolutional layer and a plurality of branch capsule networks respectively connected to the convolutional layer;

[0159] Accordingly, the second feature vector determination module 1003 includes:

[0160] A second fixed attribute feature vector determining unit, used to determine a second fixed attribute feature vector of a target object on a target image;

[0161] A second adjustable attribute feature vector determination unit is used to input the target image and the image label corresponding to the target image into the multi-branch estimator to obtain multiple second adjustable attribute feature vectors of the target object on the target image; wherein the image label is used to mark the adjustable attribute; that is, the second adjustable attribute feature vector determination unit can correspond to the multi-branch estimator in the image processing model, and the multi-branch estimator has the function of generating the second adjustable attribute feature vector of the target object on the target image;

[0162] The adjustable attribute difference determination module 1004 includes:

[0163] a vector difference value determining unit, configured to determine a vector difference value between the first adjustable attribute feature vector and each second adjustable attribute feature vector;

[0164] The adjustable attribute difference determination unit is used to determine the adjustable attribute difference based on a plurality of vector difference values.

[0165] In some embodiments, the image processing model may further include an encoder and a decoder;

[0166] Accordingly, the first feature vector determination module 1001 includes:

[0167] A first fixed attribute feature vector determination unit is used to input the sample image into the encoder to obtain a first fixed attribute feature vector of the target object; that is, the first fixed attribute feature vector determination unit may correspond to the encoder in the image processing model, and the encoder has the function of generating the first fixed attribute feature vector of the target object on the sample image;

[0168] A first adjustable attribute feature vector determination unit is used to input a modulus value predetermined according to an attribute adjustment requirement of a target object into a decoder to obtain a first adjustable attribute feature vector of the target object; that is, the first adjustable attribute feature vector determination unit may correspond to a decoder in an image processing model, and the decoder has a function of generating a first adjustable attribute feature vector of a target object on a sample image;

[0169] The second fixed attribute feature vector determining unit is specifically used for:

[0170] The target image is input into the encoder to obtain the second fixed attribute feature vector of the target object, that is, the encoder can also be used to generate the second fixed attribute feature vector of the target object on the target image.

[0171] In some embodiments, the adjustable attribute difference determination module 1004 is specifically used to:

[0172] Determine the modulus value of the second adjustable attribute feature vector, and determine the difference between the modulus value and the modulus value corresponding to the first adjustable attribute feature vector, and use the difference as the adjustable attribute difference; wherein the modulus value corresponding to the first adjustable attribute feature vector is predetermined according to the attribute adjustment requirement of the target object;

[0173] The fixed attribute difference determination module 1005 is specifically used for:

[0174] The modulus values ​​of the first fixed attribute feature vector and the second fixed attribute feature vector are determined respectively, and the difference between the two modulus values ​​is used as the fixed attribute difference degree.

[0175] In some embodiments, the image processing model training device 1000 provided by the embodiment of the present disclosure further includes:

[0176] An initial adjustable attribute feature vector acquisition module is used to acquire an initial adjustable attribute feature vector of a target object on a sample image;

[0177] A verification image generation module is used to perform vector fusion on the initial adjustable attribute feature vector and the second fixed attribute feature vector, and then input the result into a generator in a generative adversarial network to generate a verification image corresponding to the target image;

[0178] A third feature vector determination module, used to determine a third adjustable attribute feature vector of the target object on the verification image;

[0179] An adjustable attribute verification value determination module, configured to determine an adjustable attribute verification value based on the initial adjustable attribute feature vector and the third adjustable attribute feature vector;

[0180] The network parameter adjustment module 1006 is specifically used for:

[0181] Based on the adjustable attribute difference, the fixed attribute difference and the adjustable attribute verification value, the network parameters of the image processing model are adjusted.

[0182] In some embodiments, the image processing model may further include a decoder;

[0183] Accordingly, the initial adjustable attribute feature vector acquisition module includes:

[0184] An initial modulus length value acquisition unit, used to acquire an initial modulus length value corresponding to an adjustable attribute of a target object;

[0185] The initial adjustable attribute feature vector acquisition unit is used to input the initial modulus value into the decoder to obtain the initial adjustable attribute feature vector; that is, the initial adjustable attribute feature vector acquisition unit can also correspond to the decoder in the image processing model, and the decoder can also be used to generate the initial adjustable attribute feature vector of the target object on the sample image.

[0186] In some embodiments, the image processing model may further include a multi-branch estimator, the multi-branch estimator including a convolution layer and a plurality of branch capsule networks respectively connected to the convolution layer; the initial module length value acquisition unit is specifically used for:

[0187] The sample image and the image label corresponding to the sample image are input into the multi-branch estimator, and the initial modulus value corresponding to the adjustable attribute of the target object is determined according to the output result of the multi-branch estimator; that is, the initial modulus value acquisition unit can also correspond to the multi-branch estimator in the image processing model, and the multi-branch estimator can also be used to output the initial modulus value corresponding to the adjustable attribute of the target object;

[0188] The third feature vector determination module is specifically used for:

[0189] Inputting the verification image and the image label corresponding to the verification image into the multi-branch estimator to obtain a third adjustable attribute feature vector of the target object on the verification image;

[0190] Among them, image tags are used to mark adjustable attributes.

[0191] In some embodiments, the generative adversarial network further includes a multi-branch discriminator, the multi-branch discriminator and the multi-branch estimator have the same number of branches, and the multi-branch discriminator and the multi-branch estimator share a convolutional layer;

[0192] Each branch of the multi-branch discriminator is used to discriminate the target image based on the sample image.

[0193] The image processing model training device provided in the embodiment of the present disclosure can execute any image processing model training method provided in the embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. The contents not described in detail in the embodiment of the device of the present disclosure can refer to the description in any method embodiment of the present disclosure.

[0194] Fig.11 This is a schematic diagram of the structure of an image generation device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and can be integrated on any electronic device with computing capabilities, such as a terminal or a server. Furthermore, the electronic device can also be integrated on other devices with image processing requirements, such as autonomous driving vehicles, unmanned vehicles or drones, and can be deployed specifically according to actual conditions.

[0195] The image generation device provided in the embodiment of the present disclosure is implemented based on a pre-trained image processing model. The image processing model is obtained based on any image processing model training method provided in the embodiment of the present disclosure. The image processing model includes a generative adversarial network. For details, please refer to the description in the above method embodiment.

[0196] like Fig.11 As shown, the image generating device 1100 provided in the embodiment of the present disclosure includes a feature vector determining module 1101 and a target image generating module 1102, wherein:

[0197] The feature vector determination module 1101 is used to determine a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on the image to be processed; wherein the first adjustable attribute feature vector is determined according to the attribute adjustment requirement of the target object;

[0198] The target image generation module 1102 is used to perform vector fusion on the first adjustable attribute feature vector and the first fixed attribute feature vector, and input the result into a generator in a generative adversarial network to generate a target image corresponding to the image to be processed.

[0199] The image generation device provided in the embodiment of the present disclosure can execute any image generation method provided in the embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. The contents not described in detail in the embodiment of the device of the present disclosure can refer to the description in any method embodiment of the present disclosure.

[0200] Fig.12 Schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. Fig.12 As shown, the electronic device 1200 includes a central processing unit (CPU) 1201, which can perform various processes in the aforementioned embodiments according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage part 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the electronic device 1200 are also stored. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0201] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed, so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0202] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a readable medium, and the computer program includes a program code for executing any image processing model training method or image generation method provided by an embodiment of the present disclosure. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1209, and / or installed from a removable medium 1211.

[0203] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a part of a module, a program segment or a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0204] The units or modules involved in the embodiments of the present disclosure may be implemented by software or hardware. The units or modules described may also be arranged in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0205] In addition, the embodiments of the present disclosure further provide a computer-readable storage medium, which may be the computer-readable storage medium included in the device described in the above-mentioned embodiment; or may be a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and when the programs are executed by one or more processors, the electronic device implements any image processing model training method or image generation method provided in the embodiments of the present disclosure.

[0206] In summary, the embodiments of the present disclosure provide an image processing model training method, an image generation method, an apparatus, a device and a medium. In the embodiments of the present disclosure, the adjustable attributes and the fixed attributes of the target object are separated based on the attribute adjustment requirements of the target object on the sample image, and the first adjustable attribute feature vector of the target object is determined based on the attribute adjustment requirements; then, the first fixed attribute feature vector of the target object is combined and input into the generator of the generative adversarial network in the image processing model after vector fusion to obtain a target image corresponding to the sample image; finally, based on the obtained adjustable attribute feature vectors and fixed attribute feature vectors, the adjustable attribute difference and the fixed attribute difference are obtained, and the network parameters of the image processing model are adjusted. The network parameter adjustment can be performed multiple times by repeating the aforementioned operations until an image processing model that meets the image generation requirements is obtained. The disclosed embodiments solve the problem that existing models in the field of image processing can only process a limited number of discrete states of the attributes of the target object on the image, and the problem that the adjustable attributes and fixed attributes of the target object cannot be effectively separated during the model processing, and thus the state control and image generation for a certain specific attribute of the target object cannot be realized. The image processing model trained by using the technical solution of the disclosed embodiments can realize continuous processing of the attributes of the target object on the image, without being limited to processing discrete states of the attributes of the target object, and can realize state control and image generation for a certain specific attribute of the target object. For example, by controlling the state of a certain specific attribute of the target object, an image that meets the attribute adjustment requirements of the target object can be generated, thereby improving the generation efficiency of the required image.

[0207] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0208] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training an image processing model, characterized in that: The image processing model includes a generative adversarial network, and the method includes: Determine a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on a sample image; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object, the first fixed attribute feature vector is determined according to a fixed attribute of the target object, and the first adjustable attribute feature vector is determined according to an adjustable attribute of the target object, the adjustable attribute refers to an attribute that needs to be state-controlled, and the fixed attribute refers to an attribute that needs to keep its state unchanged; After vector fusion of the first adjustable attribute feature vector and the first fixed attribute feature vector, the vectors are input into a generator in the generative adversarial network to generate a target image corresponding to the sample image; Determine a second adjustable attribute feature vector and a second fixed attribute feature vector of the target object on the target image; Determining the adjustable attribute difference based on the first adjustable attribute feature vector and the second adjustable attribute feature vector, the image processing model further comprising a multi-branch estimator, the multi-branch estimator comprising a convolutional layer and a plurality of branch capsule networks respectively connected to the convolutional layer; Accordingly, determining a second adjustable attribute feature vector of the target object on the target image includes: Inputting the target image and the image label corresponding to the target image into the multi-branch estimator to obtain a plurality of second adjustable attribute feature vectors of the target object on the target image; wherein the image label is used to mark the adjustable attribute; The determining the adjustable attribute difference based on the first adjustable attribute feature vector and the second adjustable attribute feature vector includes: determining a vector difference value between the first adjustable attribute feature vector and each second adjustable attribute feature vector; Determining the adjustable attribute difference based on a plurality of the vector difference values; Determining a fixed attribute difference based on the first fixed attribute feature vector and the second fixed attribute feature vector, wherein the image processing model further includes a capsule network; Accordingly, determining the fixed attribute difference based on the first fixed attribute feature vector and the second fixed attribute feature vector includes: inputting the first fixed attribute feature vector and the second fixed attribute feature vector into the capsule network respectively, and determining the fixed attribute difference according to the output result of the capsule network; Based on the adjustable attribute difference and the fixed attribute difference, a network parameter of the image processing model is adjusted.

2. The method according to claim 1, characterized in that The image processing model also includes an encoder and a decoder; Accordingly, the step of determining a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on a sample image includes: Inputting the sample image into the encoder to obtain a first fixed attribute feature vector of the target object; Inputting a modulus value predetermined according to the property adjustment requirement of the target object into the decoder to obtain a first adjustable property feature vector of the target object; The determining a second fixed attribute feature vector of the target object on the target image comprises: The target image is input into the encoder to obtain a second fixed attribute feature vector of the target object.

3. The method according to claim 1, characterized in that The determining the adjustable attribute difference based on the first adjustable attribute feature vector and the second adjustable attribute feature vector includes: Determine a modulus value of the second adjustable attribute feature vector, and determine a difference between the modulus value and the modulus value corresponding to the first adjustable attribute feature vector, and use the difference as the adjustable attribute difference; wherein the modulus value corresponding to the first adjustable attribute feature vector is predetermined according to the attribute adjustment requirement of the target object; The determining of the fixed attribute difference based on the first fixed attribute feature vector and the second fixed attribute feature vector includes: The modulus values ​​of the first fixed attribute feature vector and the second fixed attribute feature vector are determined respectively, and the difference between the two modulus values ​​is used as the fixed attribute difference degree.

4. The method according to claim 1, characterized in that Also includes: Obtaining an initial adjustable attribute feature vector of the target object on the sample image; After vector fusion of the initial adjustable attribute feature vector and the second fixed attribute feature vector, the vectors are input into a generator in the generative adversarial network to generate a verification image corresponding to the target image; Determining a third adjustable attribute feature vector of the target object on the verification image; Determining an adjustable attribute verification value based on the initial adjustable attribute feature vector and the third adjustable attribute feature vector; The adjusting the network parameters of the image processing model based on the adjustable attribute difference and the fixed attribute difference includes: Based on the adjustable attribute difference, the fixed attribute difference and the adjustable attribute verification value, a network parameter of the image processing model is adjusted.

5. The method according to claim 4, characterized in that The image processing model also includes a decoder; Accordingly, the step of obtaining an initial adjustable attribute feature vector of the target object on the sample image includes: Obtaining an initial module length value corresponding to an adjustable attribute of the target object; The initial modulus value is input into the decoder to obtain the initial adjustable attribute feature vector.

6. The method according to claim 5, characterized in that The image processing model also includes a multi-branch estimator, which includes a convolutional layer and a plurality of branch capsule networks respectively connected to the convolutional layer; Correspondingly, the step of obtaining the initial module length value corresponding to the adjustable attribute of the target object includes: Inputting the sample image and the image label corresponding to the sample image into the multi-branch estimator, and determining the initial modulus value corresponding to the adjustable attribute of the target object according to the output result of the multi-branch estimator; The determining of a third adjustable attribute feature vector of the target object on the verification image comprises: Inputting the verification image and the image label corresponding to the verification image into the multi-branch estimator to obtain a third adjustable attribute feature vector of the target object on the verification image; The image tag is used to mark the adjustable attribute.

7. The method according to claim 1, characterized in that The generative adversarial network further includes a multi-branch discriminator, the multi-branch discriminator and the multi-branch estimator have the same number of branches, and the multi-branch discriminator and the multi-branch estimator share the convolutional layer; Each branch of the multi-branch discriminator is used to discriminate the target image according to the sample image.

8. An image generation method, characterized in that: The image generation method is implemented based on a pre-trained image processing model, the image processing model is obtained based on the image processing model training method according to any one of claims 1 to 7, the image processing model includes a generative adversarial network, and the image generation method includes: Determine a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on the image to be processed; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object; After vector fusion of the first adjustable attribute feature vector and the first fixed attribute feature vector, the vector is input into a generator in the generative adversarial network to generate a target image corresponding to the image to be processed, wherein the first fixed attribute feature vector is determined according to the fixed attribute of the target object, and the first adjustable attribute feature vector is determined according to the adjustable attribute of the target object.

9. An image processing model training device, characterized in that: The image processing model includes a generative adversarial network, and the device includes: A first feature vector determination module is used to determine a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on a sample image; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object, the first fixed attribute feature vector is determined according to a fixed attribute of the target object, and the first adjustable attribute feature vector is determined according to an adjustable attribute of the target object, wherein the adjustable attribute refers to an attribute that needs to be state-controlled, and the fixed attribute refers to an attribute that needs to keep its state unchanged; a target image generation module, configured to perform vector fusion on the first adjustable attribute feature vector and the first fixed attribute feature vector, and input the resultant vector into a generator in the generative adversarial network to generate a target image corresponding to the sample image; A second feature vector determination module, used to determine a second adjustable attribute feature vector and a second fixed attribute feature vector of the target object on the target image; An adjustable attribute difference determination module, configured to determine the adjustable attribute difference based on the first adjustable attribute feature vector and the second adjustable attribute feature vector, wherein the image processing model further comprises a multi-branch estimator, wherein the multi-branch estimator comprises a convolutional layer and a plurality of branch capsule networks respectively connected to the convolutional layer; Accordingly, the second feature vector determination module includes: A second adjustable attribute feature vector determining unit is used to input the target image and the image label corresponding to the target image into the multi-branch estimator to obtain a plurality of second adjustable attribute feature vectors of the target object on the target image; wherein the image label is used to mark the adjustable attribute; The adjustable attribute difference determination module includes: a vector difference value determining unit, configured to determine a vector difference value between the first adjustable attribute feature vector and each second adjustable attribute feature vector; An adjustable attribute difference determination unit, configured to determine the adjustable attribute difference based on a plurality of the vector difference values; a fixed attribute difference determination module, configured to determine a fixed attribute difference based on the first fixed attribute feature vector and the second fixed attribute feature vector, wherein the image processing model further comprises a capsule network; Correspondingly, the fixed attribute difference determination module is specifically used to input the first fixed attribute feature vector and the second fixed attribute feature vector into the capsule network respectively, and determine the fixed attribute difference according to the output result of the capsule network; A network parameter adjustment module is used to adjust the network parameters of the image processing model based on the adjustable attribute difference and the fixed attribute difference.

10. An image generating device, characterized in that: The image generation device is implemented based on a pre-trained image processing model, the image processing model is obtained based on the image processing model training method according to any one of claims 1 to 7, the image processing model includes a generative adversarial network, and the image generation device includes: A feature vector determination module, used to determine a first adjustable attribute feature vector and a first fixed attribute feature vector of a target object on the image to be processed; wherein the first adjustable attribute feature vector is determined according to an attribute adjustment requirement of the target object; A target image generation module is used to perform vector fusion on the first adjustable attribute feature vector and the first fixed attribute feature vector, and input the result into a generator in the generative adversarial network to generate a target image corresponding to the image to be processed, wherein the first fixed attribute feature vector is determined according to the fixed attribute of the target object, and the first adjustable attribute feature vector is determined according to the adjustable attribute of the target object.

11. An electronic device, characterized in that: It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the electronic device implements the image processing model training method described in any one of claims 1 to 7, or implements the image generation method described in claim 8.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the image processing model training method described in any one of claims 1 to 7, or implements the image generation method described in claim 8.