Image generation method, device, storage medium and computer equipment

Abstract

The embodiment of the present application provides an image generation method, apparatus, storage medium and computer equipment, including: by inputting a first segmentation mask representing different parts of a target object and a first posture image of the target object at a second viewing angle into a trained posture encoder, so as to extract posture features containing the posture of each part. And by inputting each of the appearance features and the posture features into different trained part feature generators respectively, so that different trained part feature generators respectively process the detail changes of different parts at a second viewing angle, so that the target object in the second object image finally generated has a higher consistency in appearance with the target object in the first object image, thereby improving the accuracy and consistency of the appearance of the object at different viewing angles.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, device, storage medium and computer equipment. Background Art

[0002] With the rapid advancement and widespread application of artificial intelligence (AI) technology, AI models are gradually being adopted in application scenarios such as urban security and intelligent surveillance. Object re-identification is a crucial task, whose goal is to accurately identify the same object in multiple non-overlapping cameras.

[0003] In the related art, when the AI ​​model re-identifies an object, since the object is previously treated as a whole, when the viewing angle changes, the images of the object at different angles corresponding to different cameras will have significant appearance differences, resulting in recognition failure. Therefore, the related art urgently needs to propose an image generation method to solve the above technical problems. Summary of the invention

[0004] The main purpose of the present application is to provide an image generation method, apparatus, storage medium and computer equipment, which can improve the accuracy and consistency of the appearance of an object at different viewing angles.

[0005] In a first aspect, an embodiment of the present application provides an image generation method, comprising:

[0006] Acquire a first object image, determine a first segmentation mask of a target object in the first object image, wherein the first segmentation mask is used to characterize different parts of the target object, and the first object image is an image of the target object at the first viewing angle;

[0007] Determine a first posture image of the target object at a second viewing angle;

[0008] Inputting the first segmentation mask and the first posture image into a trained posture encoder to extract posture features of the target object at the second viewing angle;

[0009] Inputting the first object image into a trained appearance encoder to extract overall appearance features of the target object, wherein the overall appearance features include appearance features corresponding to each appearance dimension;

[0010] Inputting each of the appearance features and the posture features into different trained part feature generators to obtain multiple part features;

[0011] Each of the part features is fused to obtain a fused feature, and the fused feature is input into a trained rendering network to generate a second object image, where the second object image is an image of the target object at the second viewing angle.

[0012] In a second aspect, an embodiment of the present application provides an image generating device, including:

[0013] a first determining unit, configured to acquire a first object image, determine a first segmentation mask of a target object in the first object image, wherein the first segmentation mask is used to characterize different parts of the target object, and the first object image is an image of the target object at the first viewing angle;

[0014] A second determining unit, configured to determine a first posture image of the target object at a second viewing angle;

[0015] a first extraction unit, configured to input the first segmentation mask and the first posture image into a trained posture encoder to extract posture features of the target object at the second viewing angle;

[0016] a second extraction unit, configured to input the first object image into a trained appearance encoder to extract a total appearance feature of the target object, wherein the total appearance feature includes an appearance feature corresponding to each appearance dimension;

[0017] A generating unit, used for inputting each of the appearance features and the posture features into different trained part feature generators to obtain a plurality of part features;

[0018] A fusion unit is used to fuse each of the part features to obtain a fusion feature, and input the fusion feature into a trained rendering network to generate a second object image, where the second object image is an image of the target object at the second viewing angle.

[0019] In a third aspect, an embodiment of the present application provides a storage medium, wherein the computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute any of the above image generation methods.

[0020] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above image generation methods when executing the computer program.

[0021] In an embodiment of the present application, a first segmentation mask of a target object in the first object image is determined by acquiring a first object image, wherein the first segmentation mask is used to characterize different parts of the target object, and the first object image is an image of the target object at the first viewing angle; a first posture image of the target object at a second viewing angle is determined; the first segmentation mask and the first posture image are input into a trained posture encoder to extract posture features of the target object at the second viewing angle; the first object image is input into a trained appearance encoder to extract total appearance features of the target object, wherein the total appearance features include appearance features corresponding to each appearance dimension; each of the appearance features and the posture features are respectively input into different trained part feature generators to obtain multiple part features; each of the part features is fused to obtain a fused feature, and the fused feature is input into a trained rendering network to generate a second object image, wherein the second object image is an image of the target object at the second viewing angle. Compared with the related art, in which there are significant appearance differences in images of an object at different viewing angles, the embodiment of the present application can improve the accuracy and consistency of the appearance of an object at different viewing angles.

[0022] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of a scene of an image generation system provided in an embodiment of the present application.

[0025] Figure 2 A flowchart of an image generation method provided in an embodiment of the present application.

[0026] Figure 3 A schematic diagram of the structure of a part feature generator provided in an embodiment of the present application.

[0027] Figure 4 A schematic diagram of generating images of various objects provided in an embodiment of the present application.

[0028] Figure 5Another flowchart of the image generation method provided in an embodiment of the present application.

[0029] Figure 6 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application.

[0030] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0032] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0033] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0034] Artificial Intelligence (AI) is an important part of the intelligent discipline, attempting to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial Intelligence is a very broad science, including robotics, speech recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, etc.

[0035] Segmentation mask is a representation used in image segmentation tasks. In an image, a segmentation mask is usually a two-dimensional matrix or array with the same size as the original image. The value of each element represents the category or region belonging to the corresponding pixel in the image.

[0036] An encoder is a component or algorithm that converts input data into a specific encoding or feature representation. In computer science and machine learning, the role of an encoder is to compress input information, extract key features, or perform some form of transformation for subsequent processing, analysis, or transmission. The design and function of an encoder depends on the specific application scenario and task requirements. Its purpose is to make data easier to process, classify, generate, or compare with other data in a new representation.

[0037] For example, in deep learning, an encoder is usually part of a neural network that maps input images, text, or other data to a low-dimensional feature space to achieve an effective representation of the data. This encoded feature usually contains the key information of the input data while removing some redundant or unimportant parts.

[0038] A generator is a model or component that can generate new data. The task of a generator is to create new samples that are similar to but not identical to the training data based on the learned patterns and rules. It gradually generates data with certain characteristics and structures by processing and transforming the input random noise or latent variables.

[0039] The discriminator is a classifier that receives a data sample as input and outputs a value between 0 and 1, indicating the probability estimate of the input data being the real data.

[0040] For generative adversarial networks, the discriminator and generator are continuously optimized during adversarial training. The generator strives to generate data that is as realistic as possible to deceive the discriminator, while the discriminator continuously improves its ability to distinguish between real data and generated data. Through this competitive approach, the performance of both the generator and the discriminator gradually becomes higher, and the generator is eventually able to generate very high-quality data that is difficult to distinguish from real data.

[0041] Wasserstein Generative Adversarial Networks (WGAN) is a dynamic game process between a generator and a discriminator. First, the parameters of the generator and the discriminator are randomly initialized. Then, alternating training is performed.

[0042] The process of alternating training is as follows: 1. Fix the generator and train the discriminator: input the real data and the fake data generated by the generator into the discriminator, and the discriminator updates its parameters by maximizing the probability of correctly classifying the real data and misclassifying the fake data. 2. Fix the discriminator and train the generator: The generator generates new fake data and tries to make the discriminator misclassify it as real data. The generator updates its parameters by minimizing the probability of the discriminator correctly classifying the generated data (that is, maximizing the probability of the discriminator misclassifying the generated data).

[0043] Repeat the above steps until the generator and the discriminator reach a state of equilibrium, at which point the generator can generate very realistic fake data, and the discriminator has difficulty distinguishing between real data and fake data. WGAN uses Wasserstein distance (also known as Earth-Mover distance) to measure the difference between the real data distribution and the generated data distribution, replacing the cross entropy loss in traditional GAN. Wasserstein distance has better mathematical properties, can provide more stable gradients, and contributes to the stability of training.

[0044] In addition, in order to satisfy the Lipschitz continuity condition (ensuring that the rate of change of the function is bounded), WGAN uses a simple method called weight clipping. That is, the weight of the discriminator is limited to a fixed range, such as [-0.01, 0.01]. This helps ensure that the discriminator does not become too steep, making the training more stable.

[0045] Objective,Objective,In application scenarios such as urban security and intelligent surveillance, object re-identification is a crucial task, which aims to accurately identify the same object in multiple non-overlapping,cameras.,However, due to changes in factors such as camera perspective,,lighting conditions, and object posture, existing re-identification methods often face the problem of low,recognition accuracy.,Factors such as perspective, object posture, and environmental changes cause the images of the same object under different ,cameras have significant appearance differences, which increases the difficulty of recognition.,Traditional methods mainly rely on feature extraction and,comparison under a single perspective, which cannot fully capture the detailed changes of the object,under different perspectives, which in turn affects the recognition,effect.

[0046] However, the methods in the related art generally regard the object as a whole, without considering the change patterns and laws of different parts of the object under different viewing angles.

[0047] In order to solve the above problems, the embodiment of the present application inputs the first segmentation mask representing different parts of the target object and the first posture image of the target object at the second viewing angle into the trained posture encoder to extract the posture features containing the posture of each part. And by inputting each of the appearance features and the posture features into different trained part feature generators respectively, so that different trained part feature generators can process the detail changes of different parts at the second viewing angle respectively, so that the target object in the second object image finally generated is more consistent with the target object in the first object image in appearance, thereby improving the accuracy and consistency of the appearance of the object at different viewing angles. Please continue to refer to the following specific embodiments for details.

[0048] See also Figure 1 , Figure 1 A schematic diagram of a scene of an image generation system provided in an embodiment of the present application, which includes a terminal 140, the Internet 130, a gateway 120, a computer device 110, etc.

[0049] The terminal 140 includes but is not limited to a pre-configured road camera, or an electronic device with a camera component such as a mobile phone or a tablet. In addition, it can be a single device or a collection of multiple devices. The terminal 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0050] Computer equipment refers to a computer system that can provide certain services to the terminal 140. Compared with ordinary terminals 140, computer equipment 110 has higher requirements in terms of stability, security, performance, etc. Computer equipment 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc.

[0051] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the terminal 140 to the computer device 110 must be sent to the corresponding computer device 110 through the gateway 120. The message sent by the computer device 110 to the terminal 140 must also be sent to the corresponding terminal 140 through the gateway 120.

[0052] The image generating method according to the embodiment of the present disclosure may be implemented on the computer device 110 .

[0053] It should be noted that Figure 1The scene diagram of the image generation system shown is only an example. The image generation system and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of image processing technology and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0054] In this embodiment, the description will be made from the perspective of an image generating device, which may be specifically integrated in a computer device having a storage unit and a microprocessor installed therein and having computing capabilities.

[0055] See also Figure 2 , Figure 2 A schematic diagram of a flow chart of an image generation method provided in an embodiment of the present application. The image generation method comprises:

[0056] In step 201, a first object image is acquired, and a first segmentation mask of a target object in the first object image is determined. The first segmentation mask is used to characterize different parts of the target object. The first object image is an image of the target object at a first viewing angle.

[0057] The first object image is an image obtained when capturing the target object at the first viewing angle, and the first segmentation mask is a first segmentation mask for representing different parts of the target object obtained by performing image segmentation on the first object image containing the target object. .

[0058] Specifically, the image segmentation of the first object image containing the target object can be performed in the following ways: 1. Manual annotation: The user uses an image annotation tool to manually outline the boundaries of different parts of the target object on the image to determine the category or part to which each pixel belongs. 2. Different parts of the target object are distinguished by algorithms such as threshold segmentation and edge detection. Threshold segmentation is to select one or more thresholds according to the pixel value distribution of the image to divide the pixels into different categories, thereby dividing the target object into multiple parts. Edge detection is to detect the gradient change of the gray value of the image through operators such as the Sobel operator and the Canny operator to determine the edge.

[0059] In some implementations, determining a first segmentation mask of a target object in the first object image includes:

[0060] The first object image is segmented to generate a plurality of first segmentation masks with different colors or values, each of the first segmentation masks corresponding to a part of the target object.

[0061] Among them, since the embodiment of the present application requires detailed processing for each part of the target object, when performing image segmentation on the first object image, it is necessary to use a first segmentation mask with different colors or values ​​to represent different parts, that is, each first segmentation mask corresponds to a part of the target object.

[0062] For example, when the color of the first segmentation mask is used to represent different parts, blue represents the head of the target object, green represents the torso of the target object, red represents the arm of the target object, brown represents the lower limbs of the target object, etc.; when the numerical value of the first segmentation mask is used to represent different parts, the head of the target object is represented by the numerical value 1, the torso of the target object is represented by the numerical value 2, the arm of the target object is represented by the numerical value 3, the lower limbs of the target object are represented by the numerical value 4, etc.

[0063] In this way, different parts of the target object can be represented by the first segmentation mask of different colors or values, which can clearly distinguish the various parts of the target object and provide an accurate basis for the subsequent detailed processing of each part. This allows more focus on feature extraction, analysis and operation of each part during the processing, improving the accuracy of the processing.

[0064] In step 202, a first posture image of the target object at a second viewing angle is determined.

[0065] The first posture image is the posture of the target object at the second viewing angle, such as standing posture, arm bending posture, etc. In order to ensure that the posture of the target object generated subsequently at the specified second viewing angle corresponds to the posture at the first viewing angle, it is necessary to determine the first posture image of the target object at the second viewing angle. .

[0066] In some implementations, determining a first posture image of the target object at a second viewing angle includes:

[0067] (1) converting the target object in the first object image into a three-dimensional virtual object;

[0068] (2) An image of the three-dimensional virtual object at a second viewing angle is captured to obtain a first posture image of the target object at the second viewing angle.

[0069] Among them, since the first object image is a two-dimensional image, in order to determine the first posture image of the target object at the second perspective, it is necessary to input the two-dimensional image into the existing three-dimensional grid construction model to obtain a three-dimensional grid data representation of the first object image, that is, a three-dimensional virtual object. At this time, the two-dimensional image has been converted into a three-dimensional image, and the image of the three-dimensional virtual object at the specified second perspective is collected to obtain the first posture image of the target object at the second perspective.

[0070] Specifically, the two-dimensional image is input into the existing three-dimensional grid construction model, and the processing method for obtaining the three-dimensional virtual object can be: the content in the pedestrian image is analyzed to identify the object part. The model uses methods based on deep learning, computer vision technology, etc. to estimate the body shape and posture of the object. By extracting and analyzing the features in the image, the model constructs a three-dimensional grid data representation, which describes the shape and structure of the object's body in three-dimensional space. For example, a three-dimensional grid can be composed of a series of vertices and edges connecting these vertices, roughly outlining the outline of the object.

[0071] In this way, by converting a 2D image into a 3D virtual object, more 3D information can be used to estimate the pose of the target object. Compared with pose estimation methods based only on 2D images, 3D models can provide more accurate depth information and spatial relationships, thereby improving the accuracy of pose estimation. The pose of the target object in a 2D image may have multiple interpretations, because different 3D poses may produce similar appearances when projected onto a 2D plane. By constructing a 3D virtual object, the uncertainty can be reduced and the pose of the target object in 3D space can be determined more accurately.

[0072] In step 203, the first segmentation mask and the first posture image are input into the trained posture encoder to extract posture features of the target object at the second viewing angle.

[0073] Among them, posture features is a high-dimensional vector, where each element in the vector represents a specific aspect of the target object’s posture, such as the position, angle, and direction of a body part.

[0074] Specifically, the process of performing posture feature extraction on the first segmentation mask and the first posture image by the posture encoder can refer to the following formula:

[0075] ;

[0076] in, is the first segmentation mask, is the first pose image, It is the posture feature extraction function of the posture encoder.

[0077] Specifically, the attitude encoder It is a shallow network consisting of four convolutional layers and four residual blocks. The first convolutional layer performs the first convolution operation on the first segmentation mask and the first posture image of the input. The convolutional layer extracts local features of the input data through a set of learnable filters. The filter slides on the image, multiplies and adds with the local area to generate a feature map. This process can capture low-level features in the image, such as edge features, texture features, etc. Subsequent convolutional layers continue to further extract features from the feature map obtained by the first convolutional layer to capture more complete information. Each residual block consists of multiple convolutional layers, batch normalization layers, and activation functions. The residual block is designed to solve the gradient vanishing problem in deep neural networks, making the network easier to train.

[0078] Based on the composition results of the above-mentioned posture encoder, the first segmentation mask and the first posture image are subjected to feature extraction through the first convolution layer; the features are normalized through the batch normalization layer to accelerate the training process and improve the stability of the model. Then, nonlinearity is introduced through the activation function to increase the expressive power of the posture encoder. After the above-mentioned processing, the posture features are residually connected with the input data, that is, the input data is added to the processed features. This ensures that information can flow more effectively in the network and avoid information loss. After being processed by four residual blocks, the features are continuously refined and enhanced to extract more advanced and representative posture features. After being processed by four convolutional layers and four residual blocks, the network outputs the posture features of the target object in the second perspective.

[0079] In this way, compared with the related art that treats the object as a whole and does not consider the change patterns and regularities of different parts of the object under different viewing angles, the embodiment of the present application uses a first segmentation mask to characterize different parts of the target object. The first segmentation mask is input into the posture encoder together with the first posture image, so that the posture encoder can combine the part information for more accurate posture feature extraction. For example, for human body posture estimation, different parts such as the head, torso, and limbs have specific characteristic patterns in different postures. The segmentation mask can help the encoder better identify and distinguish these parts, thereby more accurately capturing the posture details of each part.

[0080] In step 204, the first object image is input into the trained appearance encoder to extract the overall appearance features of the target object, where the overall appearance features include the appearance features corresponding to each appearance dimension.

[0081] The appearance feature may include multiple appearance dimensions, such as the color of the object's clothing, body shape, etc. By inputting the first object image into the trained appearance encoder , to encode the first object image and extract the appearance features corresponding to each appearance dimension of the target object to , to Overall appearance features .

[0082] Specifically, the process of extracting the overall appearance feature of the first object image by the appearance encoder can refer to the following formula:

[0083] ;

[0084] in, is the first object image, is the overall appearance feature extraction function of the appearance encoder.

[0085] Specifically, the appearance encoder It includes a convolution layer and a pooling layer, and four convolution layers consisting of multiple residual blocks (each residual block contains three convolution layers). The input first object image is processed through the convolution layer. A set of filters in the convolution layer slides on the image, performs multiplication and addition operations with the local area, and extracts low-level features of the image. These low-level features may include information such as edges and textures. Each filter learns a specific feature pattern, and different filters can capture different image features. The feature map processed by the convolution layer is input to the pooling layer.

[0086] The pooling layer downsamples the feature map to reduce the resolution of the feature map, thereby reducing the amount of calculation and the number of parameters. Pooling methods can include maximum pooling or average pooling. Maximum pooling selects the maximum value in a local area as the output, and average pooling calculates the average value of multiple local areas as the output. The pooling operation can retain the main feature information while removing some redundant information.

[0087] After being processed by multiple residual blocks, the features are continuously refined and enhanced. Each residual block can learn features at different levels, gradually transitioning from low-level features to high-level features. The appearance features of the target object can be fully extracted, including the appearance features of different appearance dimensions.

[0088] In step 205, each appearance feature and posture feature is input into different trained part feature generators to obtain multiple part features.

[0089] Among them, the embodiment of the present application provides a plurality of trained part feature generators, each of which is used to generate part features corresponding to each part of the target object at the second viewing angle.

[0090] For example, the part feature generator Used to generate the corresponding part features of the head , Part Feature Generator Used to generate the corresponding part features of the torso , Part Feature Generator Used to generate the corresponding part features of the arm .

[0091] Specifically, since the appearance feature includes the appearance features corresponding to different appearance dimensions, and the posture feature includes the position, angle, direction and other features of each part of the target object in the second viewing angle, each appearance feature and posture feature is input into different part feature generators, so that the generator can make full use of these two different types of information to generate part features. In this way, it can be ensured that the generated part features contain both the details of the appearance and the changes brought about by the posture.

[0092] Different parts have different characteristics and expressions in appearance and posture. By using different part feature generators to process each part separately, targeted feature generation can be performed according to the specific needs of each part.

[0093] For example, the appearance features of the head may focus more on facial features and hairstyle, while the posture features focus on the tilt angle and rotation direction of the head. For limbs, the appearance features may involve, for example, the shape and texture of the limbs, while the posture features focus on, for example, the bending degree and movement state of the joints. Using different generators can better adapt to the characteristics of different parts and generate more accurate and detailed part features.

[0094] In some implementations, each of the appearance features and the posture features are input into different trained part feature generators to obtain multiple part features, including:

[0095] Each of the appearance features and the posture features is input into different trained part feature generators to obtain multiple part features, and the adaptive weight of each pixel point in the first object image corresponding to each trained part feature generator is determined through an attention mechanism.

[0096] Among them, in order to make different trained part feature generators focus on different parts of the target object and thus perform targeted part feature generation, the focus part in the first object image corresponding to each trained part feature generator can be determined through the attention mechanism, that is, the adaptive weight of each pixel point in the area where the focus part is located is increased.

[0097] For example, the part feature generator Focusing on the head, the part feature generator The adaptive weight value of each pixel in the head area of ​​the corresponding first object image is increased to 0.9, and the adaptive weight value of each pixel in the area other than the head is decreased to 0.2, so that the part feature generator More attention is paid to the head area where pixels with higher adaptive weight values ​​are located.

[0098] Specifically, the process of extracting part features from appearance features and posture features by the part feature generator can refer to the following formula:

[0099] ;

[0100] in, Part Feature Generator Regarding posture characteristics And appearance characteristics The feature generation function used when generating part features, Part Feature Generator The generated part features are the part feature generator In the generated part features is the weight matrix composed of the adaptive weight values ​​of each pixel adjusted by the attention mechanism.

[0101] Specifically, Figure 3 As shown, Figure 3 A schematic diagram of the structure of the part feature generator provided in the embodiment of the present application. The part feature generator may be composed of 10 stylized convolution layers and 1 linear transformation layer, wherein the attention feature is output in advance in the sixth layer. Taking the first object image with a size of 128×64×3 and a posture feature with a dimension of 128×64×32 as an example, each convolution layer inputs the appearance feature The 6th and 10th convolutional layers are connected to the linear transformation layer. In the 6th convolutional layer, the linear transformation layer outputs the adaptive weight coefficient. , whose dimension is 1×64×32. In the 10th convolutional layer, the linear transformation layer outputs part features with a dimension of 512×64×32.

[0102] During the entire generation process, the first five convolutional layers mainly perform preliminary feature processing and conversion, and the sixth convolutional layer and the connected linear transformation layer begin to generate adaptive weight coefficients that guide the generation process. Subsequent convolutional layers continue to process and optimize features until the tenth convolutional layer and the connected linear transformation layer output the final part features.

[0103] In step 206, each part feature is fused to obtain a fused feature, and the fused feature is input into the trained rendering network to generate a second object image, where the second object image is an image of the target object at a second viewing angle.

[0104] In step 205, after each trained part feature generator generates a corresponding part feature, each part feature is fused to obtain a fused feature for subsequent generation of a second object image.

[0105] Specifically, the fused features are input into the trained rendering network R to generate a second object image of the target object at a second perspective. The rendering network may be composed of multiple convolutional layers with increasing resolutions. The first convolutional layer of the rendering network receives the fused features and performs preliminary feature extraction on them. The filters in the convolutional layer perform convolution operations with the fused features to capture local patterns and structural information in the fused features. The filters gradually adapt to the features of the target object during the learning process and extract important features related to image generation. After the convolution operation, a feature map is obtained. This feature map is a two-dimensional array, in which each element represents the feature value of the input fused feature at a specific position and a specific filter response. As the resolution of the convolutional layer in the rendering network increases, the details and clarity of the image gradually increase. The low-resolution convolutional layer extracts more global and abstract features, while the high-resolution convolutional layer can capture finer local details.

[0106] By gradually increasing the resolution of convolutional layers, the image details of the target object in the second perspective can be gradually restored from the fused features. The output of each convolutional layer not only contains the features extracted by the current layer, but also incorporates the information of the previous convolutional layer. This feature transfer and fusion enables the network to comprehensively utilize features at different levels and gradually build an image from global structure to local details. Finally, after being processed by multiple convolutional layers with increasing resolution, the rendering network outputs a high-resolution feature map, which can be regarded as an image representation of the target object in the second perspective. Through operations such as activation functions and normalization, the feature map can be converted into actual image pixel values ​​to generate the final second object image.

[0107] In some embodiments, the fusing of each of the site features to obtain a fused feature comprises:

[0108] (1) determining a part feature weight of each of the part features based on an adaptive weight of each pixel in each of the first object images;

[0109] (2) calculating the Hadamard product of each of the part features and the corresponding part feature weight to obtain a plurality of part features to be fused;

[0110] (3) Determine the sum of the features of the multiple parts to be fused to obtain a fusion feature.

[0111] Among them, since the adaptive weight of each pixel point in the first object image corresponding to each trained part feature generator is different, the part feature weight of the part feature generated by each trained part feature generator can be determined according to the adaptive weight of each pixel point in each first object image. The weight is in matrix form, and the part feature is also in matrix form. Therefore, when fusing multiple part features, it is necessary to calculate the Hadamard product of each part feature and the corresponding part feature weight separately to obtain multiple part features to be fused, and calculate the sum of the multiple part features to be fused to obtain the fused feature.

[0112] Specifically, the method for determining the fusion feature can refer to the following formula:

[0113]

[0114] in, Part Feature Generator The part feature weights of the corresponding part features are calculated in sequence from The sum of the product of the part feature corresponding to each part feature generator to N and the part feature weight is used to obtain the fusion feature .

[0115] In this way, by calculating the Hadamard product of each part feature and the corresponding part feature weight, the fine fusion of part features can be achieved. The Hadamard product is an element-by-element multiplication operation, which can adjust each element of the part feature according to the weight. In this way, the fused part features to be fused can more accurately reflect the importance and feature contribution of each part. The detailed information of the part features can be retained during the fusion process. Compared with simple weighted summation, the Hadamard product can better consider the local structure of the part features and the relationship between the elements, thereby reducing the loss of information during the fusion process and improving the quality of the fused features.

[0116] In some implementations, determining the part feature weight of each part feature based on the adaptive weight of each pixel in each first object image includes:

[0117] (1.1) Determine the exponential mapping value of the adaptive weight of each pixel;

[0118] (1.2) calculating the sum of the exponential mapping values ​​of each pixel in each of the first object images to obtain the exponential mapping sum value of each of the first object images;

[0119] (1.3) Determine a ratio of the adaptive weight of each pixel point in each of the first object images to the corresponding exponential mapping and value, and obtain a part feature weight of each of the part features.

[0120] The method for determining the part feature weight of each part feature may refer to the following formula:

[0121] ;

[0122] in, is the first object image OK, is the first object image Column, that is, the first object image Line Since each pixel in the first object image corresponds to an adaptive weight, the exponential mapping value of the adaptive weight of each pixel can be calculated , and calculate the sum of the exponential mapping values ​​of each pixel in each first object image, that is, the exponential mapping sum value Thus, by determining the ratio of the adaptive weight of each pixel point in each of the first object images to the corresponding exponential mapping and value, the proportion of the exponential mapping values ​​of the multiple pixels in the corresponding first object image is obtained, that is, the part feature weight of each part feature.

[0123] In this way, by determining the exponential mapping value of the adaptive weight of each pixel, the difference in weights between different pixels can be further amplified. For those pixels that are important in the image, their weights will become larger after exponential mapping, so that they will receive more attention in subsequent calculations. This helps to highlight the key parts and features in the image and improve the accuracy of the weights of the part features.

[0124] In some embodiments, the image generation method is applied to a trained image generation model D, the image generation model includes a trained pose encoder, a trained appearance encoder, a trained rendering network, and each trained part feature generator, and after inputting the fused feature into the trained rendering network to generate a second object image, further includes:

[0125] (1) determining a second posture image of the target object at the first viewing angle;

[0126] (2) generating a third object image based on the first object image and the second posture image;

[0127] (3) generating a fourth object image based on the second object image and the second posture image;

[0128] (4) constructing an image generation loss according to the first object image, the second object image, the third object image, and the fourth object image;

[0129] (5) performing authenticity evaluation on the first object image, the second object image, the third object image, and the fourth object image through the trained discriminator to construct a discriminator loss;

[0130] (6) If the image generation loss does not satisfy the first preset convergence condition and / or the discriminator loss does not satisfy the second preset convergence condition, the trained image generation model and the trained discriminator are adjusted according to the image generation loss and the discriminator loss, respectively, and the step of acquiring the first object image and determining the first segmentation mask of the target object in the first object image is returned to be executed until the image generation loss satisfies the first preset convergence condition and the discriminator loss satisfies the second preset convergence condition.

[0131] Among them, in order to further improve the authenticity of the images generated by the trained image generation model, a trained discriminator D can be configured, and a generative adversarial network is performed on the trained image generation model and the trained discriminator to further improve the authenticity of the images generated by the trained image generation model.

[0132] For details, please refer to Figure 4 , Figure 4 This is a schematic diagram of generating images of various objects provided in the embodiment of the present application. For the first object image, first determine the second posture image of the target object under the first viewing angle, that is, the posture image of the target object without changing the viewing angle. That is, input the trained image generation model G , so that the trained image generation model can generate the first object image and the second posture image generates a third object image , the second object image is the input to the trained image generation model , to generate an object image of the target object at the second viewing angle. And restore the fourth object image of the target object at the first viewing angle through the generated second object image, that is, input the trained image generation model , to obtain the fourth object image .

[0133] Thus, according to the first object image, the second object image, the third object image and the fourth object image, an image generation loss is constructed, and the authenticity of the first object image, the second object image, the third object image and the fourth object image is evaluated by the trained discriminator to construct a discriminator loss. If the image generation loss does not meet the first preset convergence condition and\or the discriminator loss does not meet the second preset convergence condition, it means that the authenticity of the object image generated by the trained image generation model is low and\or the discriminator's discrimination ability is not accurate enough. Therefore, it is necessary to adjust the trained image generation model and the network parameters of the trained discriminator according to the image generation loss and the discriminator loss respectively, and return to execute the step of obtaining the first object image and determining the first segmentation mask of the target object in the first object image, and repeat this until the image generation loss meets the first preset convergence condition and the discriminator loss meets the second preset convergence condition. At this time, the object image generated by the trained image generation model has high authenticity and the discriminator's discrimination ability is also accurate enough, then the training process of the generative adversarial network ends.

[0134] In this way, by constructing image generation loss and discriminator loss, and training the trained image generation model and the trained discriminator in the form of a generative adversarial network, the generated image gradually approaches the characteristics and distribution of the real image, thereby improving the quality and authenticity of the generated image.

[0135] In some implementations, constructing an image generation loss according to the first object image, the second object image, the third object image, and the fourth object image includes:

[0136] (4.1) determining a reconstruction loss according to a first reconstruction error between the first object image and the third object image and a second reconstruction error between the first object image and the fourth object image;

[0137] (4.2) determining an appearance feature consistency loss based on a first feature difference between a first overall appearance feature of the first object image and a second overall appearance feature of the second object image, and a second feature difference between the first overall appearance feature of the first object image and a third overall appearance feature of the fourth object image;

[0138] (4.3) determining a segmentation mask consistency loss according to a first segmentation mask error between a first segmentation mask of the first object image and a second segmentation mask of the third object image, and a second segmentation mask error between the first segmentation mask of the first object image and a third segmentation mask of the fourth object image;

[0139] (4.4) determining a first adversarial loss based on the second object image and a fourth segmentation mask of the second object image;

[0140] (4.5) determining a second adversarial loss based on the third object image and the second segmentation mask;

[0141] (4.6) determining a third adversarial loss based on the fourth object image and the third segmentation mask;

[0142] (4.7) Constructing an image generation loss based on the reconstruction loss, the appearance feature consistency loss, the segmentation mask consistency loss, the first adversarial loss, the second adversarial loss, and the third adversarial loss.

[0143] Among them, the image generation loss includes the following reconstruction loss , appearance feature consistency loss , segmentation mask consistency loss , and three adversarial losses , and .

[0144] Among them, the determination method of reconstruction loss can refer to the following formula:

[0145] ;

[0146] The reconstruction loss is used to measure the reconstruction error between the first object image and the generated third object image and the third object image. During the training process, the reconstruction loss is minimized to make the generated image as close to the original image as possible, thereby ensuring the quality of the generated image.

[0147] Specifically, That is, the first reconstruction error between the first object image and the third object image, is the second reconstruction error between the first object image and the fourth object image, and the expected value of the sum of the first reconstruction error and the second reconstruction error is the reconstruction loss, where: Represents an expectation operation.

[0148] The determination method of appearance feature consistency loss can refer to the following formula:

[0149] ;

[0150] Among them, the appearance feature consistency loss is used to measure the consistency of appearance features of the first object image with the second object image and the fourth object image. By comparing the appearance features of the first object image and the second object image after passing through the appearance encoder, as well as the appearance features of the first object image and the fourth object image after passing through the appearance encoder, it is ensured that the generated second object image and the fourth object image are similar to the first object image in appearance features.

[0151] Specifically, That is, the first feature difference between the first overall appearance feature of the first object image and the second overall appearance feature of the second object image, That is, the second feature difference between the first total appearance feature of the first object image and the third total appearance feature of the fourth object image. The expected value of the sum of the first feature difference and the second feature difference is calculated, which is the appearance feature consistency loss.

[0152] The determination method of the segmentation mask consistency loss can refer to the following formula:

[0153] ;

[0154] The segmentation mask consistency loss is used to measure the consistency of the segmentation masks of the first object image and the third object image and the fourth object image, so as to ensure that the segmentation mask of the generated image is as similar as possible to the segmentation mask of the original image.

[0155] Specifically, That is, the first segmentation mask error between the first segmentation mask of the first object image and the second segmentation mask of the third object image, That is, the second segmentation mask error between the first segmentation mask of the first object image and the third segmentation mask of the fourth object image. The expected value of the sum of the first segmentation mask error and the second segmentation mask error is calculated, which is the segmentation mask consistency loss.

[0156] The method for determining the countermeasure loss can refer to the following formula:

[0157] ;

[0158] ;

[0159] ;

[0160] in, When the second object image generated by the image generation model and the corresponding fourth segmentation mask are discriminated by the trained discriminator, it is expected that the trained discriminator considers them to be real; When the third object image generated by the image generation model and the corresponding second segmentation mask are discriminated by the trained discriminator, it is expected that the trained discriminator considers them to be real; When the fourth object image generated by the image generation model and the corresponding third segmentation mask are discriminated by the trained discriminator, it is expected that the trained discriminator considers them to be real.

[0161] Based on this, after obtaining the reconstruction loss, appearance feature consistency loss, segmentation mask consistency loss, first adversarial loss, second adversarial loss and third adversarial loss, the image generation loss is obtained by assigning different weights to each loss function and performing weighted summation of the loss functions.

[0162] In this way, by constructing the image generation loss, the trained pose encoder, the trained appearance encoder, the trained rendering network and each trained part feature generator in the trained image generation model can be strengthened to improve the accuracy of each module.

[0163] In some implementations, performing authenticity evaluation on the first object image, the second object image, the third object image, and the fourth object image through a discriminator to construct a discriminator loss includes:

[0164] (5.1) Obtaining a first output expectation of the trained discriminator for the first object image and the first segmentation mask, a second output expectation of the trained discriminator for the second object image and a fourth segmentation mask of the second object image, a third output expectation of the trained discriminator for the third object image and the second segmentation mask, and a fourth output expectation of the trained discriminator for the fourth object image and the third segmentation mask;

[0165] (5.2) determining a first discriminator loss according to a difference between the second output expectation and the first output expectation;

[0166] (5.3) determining a second discriminator loss according to a difference between the third output expectation and the first output expectation;

[0167] (5.4) determining a third discriminator loss according to a difference between the fourth output expectation and the first output expectation;

[0168] (5.5) Construct a discriminator loss based on the first discriminator loss, the second discriminator loss, and the third discriminator loss.

[0169] Among them, the discriminator loss includes the first discriminator loss , the second discriminator loss and the third discriminator loss .

[0170] Among them, the determination method of the first discriminator loss can refer to the following formula:

[0171] ;

[0172] in, is the gradient penalty term specified in the generative adversarial network, is the weight coefficient of the gradient penalty term. The weighted sum of each loss function constitutes the final loss function during training. The weight can be specified by the user according to the data set as a hyperparameter. is the second output expectation of the trained discriminator for the second object image and the fourth segmentation mask of the second object image, is the first output expectation of the trained discriminator for the first object image and the first segmentation mask, and the first discriminator loss is determined according to the difference between the second output expectation and the first output expectation , To compute the expectation under the generated data distribution, To calculate the expectation under the real data distribution.

[0173] The second discriminator loss can be determined by referring to the following formula:

[0174] ;

[0175] in, is a third output expectation of the trained discriminator for the third object image and the second segmentation mask, and a second discriminator loss is determined according to the difference between the third output expectation and the first output expectation. .

[0176] The third discriminator loss can be determined by referring to the following formula:

[0177] ;

[0178] in, is a fourth output expectation of the trained discriminator for the fourth object image and the third segmentation mask, and a third discriminator loss is determined according to the difference between the fourth output expectation and the first output expectation. .

[0179] Based on this, after obtaining the first discriminator loss, the second discriminator loss and the third discriminator loss, the discriminator loss is obtained by assigning different weights to each loss function and performing weighted summation of the loss functions.

[0180] In this way, by constructing the discriminator loss, the trained discriminator can be strengthened to improve the accuracy of the discriminator's discrimination.

[0181] As can be seen from the above, the embodiment of the present application obtains a first object image, determines a first segmentation mask of the target object in the first object image, the first segmentation mask is used to characterize different parts of the target object, and the first object image is the image of the target object at the first viewing angle; determines a first posture image of the target object at a second viewing angle; inputs the first segmentation mask and the first posture image into a trained posture encoder to extract posture features of the target object at the second viewing angle; inputs the first object image into a trained appearance encoder to extract total appearance features of the target object, the total appearance features include appearance features corresponding to each appearance dimension; inputs each of the appearance features and the posture features into different trained part feature generators respectively to obtain multiple part features; fuses each of the part features to obtain a fused feature, and inputs the fused feature into a trained rendering network to generate a second object image, the second object image is the image of the target object at the second viewing angle.

[0182] Thus, the first segmentation mask representing different parts of the target object and the first posture image of the target object at the second viewing angle are input into the trained posture encoder to extract the posture feature containing the posture of each part. And each of the appearance features and the posture features are respectively input into different trained part feature generators, so that different trained part feature generators process the detail changes of different parts at the second viewing angle respectively, so that the target object in the second object image finally generated has a higher consistency in appearance with the target object in the first object image, thereby improving the accuracy and consistency of the appearance of the object at different viewing angles.

[0183] For details, please refer to Figure 5 , Figure 5 Another flow chart of the image generation method provided by the embodiment of the present application. When a first object image is obtained, the first object image is input into the trained appearance encoder to obtain , to The first segmentation mask used to characterize different parts of the target object in the first object image is obtained, and a 3D model is generated from the first object image, and a first posture image under a second perspective is obtained by adjusting the perspective, and the first segmentation mask and the first posture image are input into the trained posture encoder to extract the posture feature. , so that the appearance features corresponding to each appearance dimension are combined with the posture features Input to different trained part feature generators to obtain the part features generated by each trained part feature generator and the adaptive weight corresponding to each pixel point in each first object image (for example Figure 5 Medium adaptive weight In order to give a higher weight to the pixels constituting the upper body of the target object in the first object image, the adaptive weight In order to give a higher weight to the pixels constituting the lower body of the target object in the first object image, the adaptive weight In order to give a higher weight to the pixel points of the arm part constituting the target object in the first object image, each part feature and the adaptive weight corresponding to each pixel point are input into the fusion module to obtain the fusion feature , the fusion features Input into the rendering network to obtain the generated second object image, and obtain the segmentation masks of the two object images, and send the real first object image and its corresponding first segmentation mask and the segmentation mask of the second object image to the trained discriminator to obtain the discrimination result through the trained discriminator, so as to perform the generative adversarial network training process on the trained image generation model and the trained discriminator.

[0184] The specific implementation of the above steps can be found in the previous embodiments, which will not be described in detail here.

[0185] In order to better implement the image generation method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above image generation method. The meanings of the terms are the same as those in the above image generation method, and the specific implementation details can refer to the description in the method embodiment.

[0186] See also Figure 6 , Figure 6 The schematic diagram of the structure of the image generation device provided in the embodiment of the present application is applied to a computer device. The image generation device may include a first determination unit 601, a second determination unit 602, a first extraction unit 603, a second extraction unit 604, a generation unit 605, and a fusion unit 606.

[0187] A first determining unit 601 is configured to acquire a first object image and determine a first segmentation mask of a target object in the first object image, wherein the first segmentation mask is used to characterize different parts of the target object, and the first object image is an image of the target object at the first viewing angle;

[0188] A second determining unit 602, configured to determine a first posture image of the target object at a second viewing angle;

[0189] A first extraction unit 603, configured to input the first segmentation mask and the first posture image into a trained posture encoder to extract posture features of the target object at the second viewing angle;

[0190] A second extraction unit 604 is used to input the first object image into the trained appearance encoder to extract the overall appearance feature of the target object, where the overall appearance feature includes the appearance feature corresponding to each appearance dimension;

[0191] A generating unit 605, configured to input each of the appearance features and the posture features into different trained part feature generators to obtain a plurality of part features;

[0192] The fusion unit 606 is used to fuse each of the part features to obtain a fused feature, and input the fused feature into the trained rendering network to generate a second object image, where the second object image is an image of the target object at the second viewing angle.

[0193] In some embodiments, the generating unit 605 is used to:

[0194] Inputting each of the appearance features and the posture features into different trained part feature generators respectively to obtain multiple part features, and determining the adaptive weight of each pixel point in the first object image corresponding to each trained part feature generator through an attention mechanism;

[0195] In some embodiments, the fusion unit 606 is used to:

[0196] Determining a part feature weight of each of the part features based on an adaptive weight of each pixel in each of the first object images;

[0197] Calculating the Hadamard product of each of the part features and the corresponding part feature weight to obtain a plurality of part features to be fused;

[0198] The sum of the features of the plurality of parts to be fused is determined to obtain a fusion feature.

[0199] In some embodiments, the generating unit 605 is used to:

[0200] Determine the exponential mapping value of the adaptive weight of each pixel;

[0201] Calculating the sum of the exponential mapping values ​​of each pixel in each of the first object images to obtain the exponential mapping sum value of each of the first object images;

[0202] The ratio of the adaptive weight of each pixel point in each of the first object images to the corresponding exponential mapping and value is determined to obtain a part feature weight of each of the part features.

[0203] In some embodiments, the first determining unit 601 is configured to:

[0204] The first object image is segmented to generate a plurality of first segmentation masks with different colors or values, each of the first segmentation masks corresponding to a part of the target object.

[0205] In some embodiments, the second determining unit 602 is configured to:

[0206] converting the target object in the first object image into a three-dimensional virtual object;

[0207] An image of the three-dimensional virtual object at a second viewing angle is acquired to obtain a first posture image of the target object at the second viewing angle.

[0208] In some embodiments, the image generation device is applied to a trained image generation model, wherein the image generation model includes a trained pose encoder, a trained appearance encoder, a trained rendering network, and each trained part feature generator, and the image generation device is further used to:

[0209] Determine a second posture image of the target object at the first viewing angle;

[0210] generating a third object image based on the first object image and the second posture image;

[0211] generating a fourth object image based on the second object image and the second posture image;

[0212] constructing an image generation loss according to the first object image, the second object image, the third object image, and the fourth object image;

[0213] Performing authenticity evaluation on the first object image, the second object image, the third object image, and the fourth object image through the trained discriminator to construct a discriminator loss;

[0214] If the image generation loss does not satisfy the first preset convergence condition and\or the discriminator loss does not satisfy the second preset convergence condition, the trained image generation model and the trained discriminator are adjusted according to the image generation loss and the discriminator loss, and the steps of acquiring the first object image and determining the first segmentation mask of the target object in the first object image are returned to execute until the image generation loss satisfies the first preset convergence condition and the discriminator loss satisfies the second preset convergence condition.

[0215] In some embodiments, the image generating device is further used to:

[0216] determining a reconstruction loss according to a first reconstruction error between the first object image and the third object image and a second reconstruction error between the first object image and the fourth object image;

[0217] determining an appearance feature consistency loss according to a first feature difference between a first overall appearance feature of the first object image and a second overall appearance feature of the second object image, and a second feature difference between the first overall appearance feature of the first object image and a third overall appearance feature of the fourth object image;

[0218] determining a segmentation mask consistency loss according to a first segmentation mask error between a first segmentation mask of the first object image and a second segmentation mask of the third object image, and a second segmentation mask error between the first segmentation mask of the first object image and a third segmentation mask of the fourth object image;

[0219] determining a first adversarial loss based on the second object image and a fourth segmentation mask of the second object image;

[0220] determining a second adversarial loss based on the third object image and the second segmentation mask;

[0221] Determining a third adversarial loss based on the fourth object image and the third segmentation mask;

[0222] An image generation loss is constructed based on the reconstruction loss, the appearance feature consistency loss, the segmentation mask consistency loss, the first adversarial loss, the second adversarial loss, and the third adversarial loss.

[0223] In some embodiments, the image generating device is further used to:

[0224] Obtaining a first output expectation of the trained discriminator for the first object image and the first segmentation mask, a second output expectation of the trained discriminator for the second object image and a fourth segmentation mask of the second object image, a third output expectation of the trained discriminator for the third object image and the second segmentation mask, and a fourth output expectation of the trained discriminator for the fourth object image and the third segmentation mask;

[0225] Determining a first discriminator loss according to a difference between the second output expectation and the first output expectation;

[0226] Determining a second discriminator loss according to a difference between the third output expectation and the first output expectation;

[0227] Determining a third discriminator loss according to a difference between the fourth output expectation and the first output expectation;

[0228] A discriminator loss is constructed based on the first discriminator loss, the second discriminator loss, and the third discriminator loss.

[0229] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0230] As can be seen from the above, the embodiment of the present application obtains a first object image through a first determination unit 601, determines a first segmentation mask of a target object in the first object image, the first segmentation mask is used to characterize different parts of the target object, and the first object image is an image of the target object at the first viewing angle; a second determination unit 602 determines a first posture image of the target object at the second viewing angle; a first extraction unit 603 inputs the first segmentation mask and the first posture image into a trained posture encoder to extract posture features of the target object at the second viewing angle; a second extraction unit 604 inputs the first object image into a trained appearance encoder to extract a total appearance feature of the target object, the total appearance feature including an appearance feature corresponding to each appearance dimension; a generation unit 605 inputs each of the appearance features and the posture features into different trained part feature generators respectively to obtain multiple part features; a fusion unit 606 fuses each of the part features to obtain a fused feature, and inputs the fused feature into a trained rendering network to generate a second object image, the second object image being an image of the target object at the second viewing angle.

[0231] Thus, the first segmentation mask representing different parts of the target object and the first posture image of the target object at the second viewing angle are input into the trained posture encoder to extract the posture feature containing the posture of each part. And each of the appearance features and the posture features are respectively input into different trained part feature generators, so that different trained part feature generators process the detail changes of different parts at the second viewing angle respectively, so that the target object in the second object image finally generated has a higher consistency in appearance with the target object in the first object image, thereby improving the accuracy and consistency of the appearance of the object at different viewing angles.

[0232] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0233] Reference Figure 7 , Figure 7The block diagram of the structure of part of the computer device 110 for implementing the embodiment of the present disclosure. The computer device 110 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 622 (for example, one or more processors) and memory 632, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 642 or data 644. Among them, the memory 632 and the storage medium 630 can be temporary storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server 600. Furthermore, the central processing unit 622 can be configured to communicate with the storage medium 630 and execute a series of instruction operations in the storage medium 630 on the server 600.

[0234] The computer device 110 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0235] The central processor 622 in the computer device 110 may be used to execute the image generation method of the embodiment of the present disclosure, for example:

[0236] Acquire a first object image, determine a first segmentation mask of a target object in the first object image, wherein the first segmentation mask is used to characterize different parts of the target object, and the first object image is an image of the target object at the first viewing angle;

[0237] Determine a first posture image of the target object at a second viewing angle;

[0238] Inputting the first segmentation mask and the first posture image into a trained posture encoder to extract posture features of the target object at the second viewing angle;

[0239] Inputting the first object image into a trained appearance encoder to extract overall appearance features of the target object, wherein the overall appearance features include appearance features corresponding to each appearance dimension;

[0240] Inputting each of the appearance features and the posture features into different trained part feature generators to obtain multiple part features;

[0241] Each of the part features is fused to obtain a fused feature, and the fused feature is input into a trained rendering network to generate a second object image, where the second object image is an image of the target object at the second viewing angle.

[0242] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the image generating methods of the aforementioned embodiments.

[0243] The present disclosure also provides a computer program product, which includes a computer program. The processor of a computer device reads and executes the computer program, so that the computer device executes the above-mentioned image generation method. For example:

[0244] Acquire a first object image, determine a first segmentation mask of a target object in the first object image, wherein the first segmentation mask is used to characterize different parts of the target object, and the first object image is an image of the target object at the first viewing angle;

[0245] Determine a first posture image of the target object at a second viewing angle;

[0246] Inputting the first segmentation mask and the first posture image into a trained posture encoder to extract posture features of the target object at the second viewing angle;

[0247] Inputting the first object image into a trained appearance encoder to extract overall appearance features of the target object, wherein the overall appearance features include appearance features corresponding to each appearance dimension;

[0248] Inputting each of the appearance features and the posture features into different trained part feature generators to obtain multiple part features;

[0249] Each of the part features is fused to obtain a fused feature, and the fused feature is input into a trained rendering network to generate a second object image, where the second object image is an image of the target object at the second viewing angle.

[0250] In addition, the terms "comprises" and "includes" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product or apparatus.

[0251] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0252] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0253] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0254] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0255] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0256] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0257] It should also be understood that the various implementations provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0258] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0259] The above is a specific description of the implementation method of the present application, but the present application is not limited to the above-mentioned implementation method. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. An image generation method, characterized in that: include: Acquire a first object image, determine a first segmentation mask of a target object in the first object image, wherein the first segmentation mask is used to characterize different parts of the target object, and the first object image is an image of the target object at a first viewing angle; Determine a first posture image of the target object at a second viewing angle; Inputting the first segmentation mask and the first posture image into a trained posture encoder to extract posture features of the target object at the second viewing angle; Inputting the first object image into a trained appearance encoder to extract overall appearance features of the target object, wherein the overall appearance features include appearance features corresponding to each appearance dimension; Inputting each of the appearance features and the posture features into different trained part feature generators to obtain multiple part features; Each of the part features is fused to obtain a fused feature, and the fused feature is input into a trained rendering network to generate a second object image, where the second object image is an image of the target object at the second viewing angle.

2. The image generation method according to claim 1, characterized in that: The method further comprises inputting each of the appearance features and the posture features into different trained part feature generators to obtain a plurality of part features, including: Inputting each of the appearance features and the posture features into different trained part feature generators respectively to obtain multiple part features, and determining the adaptive weight of each pixel point in the first object image corresponding to each trained part feature generator through an attention mechanism; The fusing of each of the part features to obtain a fused feature includes: Determining a part feature weight of each of the part features based on an adaptive weight of each pixel in each of the first object images; Calculating the Hadamard product of each of the part features and the corresponding part feature weight to obtain a plurality of part features to be fused; The sum of the features of the plurality of parts to be fused is determined to obtain a fusion feature.

3. The image generation method according to claim 2, characterized in that: The determining the part feature weight of each part feature based on the adaptive weight of each pixel in each of the first object images comprises: Determine the exponential mapping value of the adaptive weight of each pixel; Calculating the sum of the exponential mapping values ​​of each pixel in each of the first object images to obtain the exponential mapping sum value of each of the first object images; The ratio of the adaptive weight of each pixel point in each of the first object images to the corresponding exponential mapping and value is determined to obtain a part feature weight of each of the part features.

4. The image generation method according to any one of claims 1 to 3, characterized in that: The determining a first segmentation mask of the target object in the first object image comprises: The first object image is segmented to generate a plurality of first segmentation masks with different colors or values, each of the first segmentation masks corresponding to a part of the target object.

5. The image generation method according to any one of claims 1 to 3, characterized in that: The determining a first posture image of the target object at a second viewing angle includes: converting the target object in the first object image into a three-dimensional virtual object; An image of the three-dimensional virtual object at a second viewing angle is acquired to obtain a first posture image of the target object at the second viewing angle.

6. The image generation method according to claim 1, characterized in that: The method is applied to a trained image generation model, wherein the image generation model includes a trained pose encoder, a trained appearance encoder, a trained rendering network, and each trained part feature generator, and after inputting the fused features into the trained rendering network to generate a second object image, further includes: Determining a second posture image of the target object at the first viewing angle; generating a third object image based on the first object image and the second posture image; generating a fourth object image based on the second object image and the second posture image; constructing an image generation loss according to the first object image, the second object image, the third object image, and the fourth object image; Performing authenticity evaluation on the first object image, the second object image, the third object image, and the fourth object image through the trained discriminator to construct a discriminator loss; If the image generation loss does not satisfy the first preset convergence condition and\or the discriminator loss does not satisfy the second preset convergence condition, the trained image generation model and the trained discriminator are adjusted according to the image generation loss and the discriminator loss, and the steps of acquiring the first object image and determining the first segmentation mask of the target object in the first object image are returned to execute until the image generation loss satisfies the first preset convergence condition and the discriminator loss satisfies the second preset convergence condition.

7. The image generation method according to claim 6, characterized in that: The constructing an image generation loss according to the first object image, the second object image, the third object image, and the fourth object image includes: determining a reconstruction loss according to a first reconstruction error between the first object image and the third object image and a second reconstruction error between the first object image and the fourth object image; determining an appearance feature consistency loss according to a first feature difference between a first overall appearance feature of the first object image and a second overall appearance feature of the second object image, and a second feature difference between the first overall appearance feature of the first object image and a third overall appearance feature of the fourth object image; determining a segmentation mask consistency loss according to a first segmentation mask error between a first segmentation mask of the first object image and a second segmentation mask of the third object image, and a second segmentation mask error between the first segmentation mask of the first object image and a third segmentation mask of the fourth object image; determining a first adversarial loss based on the second object image and a fourth segmentation mask of the second object image; determining a second adversarial loss based on the third object image and the second segmentation mask; Determining a third adversarial loss based on the fourth object image and the third segmentation mask; An image generation loss is constructed based on the reconstruction loss, the appearance feature consistency loss, the segmentation mask consistency loss, the first adversarial loss, the second adversarial loss, and the third adversarial loss.

8. The image generation method according to claim 7, characterized in that: The performing authenticity evaluation on the first object image, the second object image, the third object image, and the fourth object image through the trained discriminator to construct a discriminator loss includes: Obtaining a first output expectation of the trained discriminator for the first object image and the first segmentation mask, a second output expectation of the trained discriminator for the second object image and a fourth segmentation mask of the second object image, a third output expectation of the trained discriminator for the third object image and the second segmentation mask, and a fourth output expectation of the trained discriminator for the fourth object image and the third segmentation mask; Determining a first discriminator loss according to a difference between the second output expectation and the first output expectation; Determining a second discriminator loss according to a difference between the third output expectation and the first output expectation; Determining a third discriminator loss according to a difference between the fourth output expectation and the first output expectation; A discriminator loss is constructed based on the first discriminator loss, the second discriminator loss, and the third discriminator loss.

9. An image generating device, characterized in that: include: a first determining unit, configured to acquire a first object image, determine a first segmentation mask of a target object in the first object image, wherein the first segmentation mask is used to characterize different parts of the target object, and the first object image is an image of the target object at a first viewing angle; A second determining unit, configured to determine a first posture image of the target object at a second viewing angle; a first extraction unit, configured to input the first segmentation mask and the first posture image into a trained posture encoder to extract posture features of the target object at the second viewing angle; a second extraction unit, configured to input the first object image into a trained appearance encoder to extract a total appearance feature of the target object, wherein the total appearance feature includes an appearance feature corresponding to each appearance dimension; A generating unit, used for inputting each of the appearance features and the posture features into different trained part feature generators to obtain a plurality of part features; A fusion unit is used to fuse each of the part features to obtain a fusion feature, and input the fusion feature into a trained rendering network to generate a second object image, where the second object image is an image of the target object at the second viewing angle.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the image generating method according to any one of claims 1 to 8.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image generating method according to any one of claims 1 to 8 is implemented.

Images