Image processing method, model generation method, and device

Abstract

Embodiments of the present application provide an image processing method, a model generation method and equipment. In the embodiments, camera pose information and at least one facial attribute feature are determined; a generator in a generative adversarial network is used to construct voxel coordinate information in a three-dimensional space based on the camera pose information; voxel features are constructed based on the voxel coordinate information and the at least one facial attribute feature; the voxel features are converted into pixel features corresponding to a two-dimensional space, and a target facial image is rendered based on the pixel features. The technical solution provided in the embodiments of the present application guarantees the image generation effect, thereby improving the image generation accuracy.

Description

Technical Field

[0001] This application relates to the field of computer image technology, and in particular to an image processing method, a model generation method, and an apparatus. Background Technology

[0002] In practical applications, there is a need to generate images of specific objects, such as facial images. How to generate more accurate images has become a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0003] This application provides an image processing method, a model generation method, and an apparatus to solve the technical problem of low image generation accuracy in the prior art.

[0004] In a first aspect, embodiments of this application provide an image processing method, including:

[0005] Determine the camera pose information and at least one facial attribute feature;

[0006] Using a generator in a generative adversarial network, voxel coordinate information in three-dimensional space is constructed based on the camera pose information;

[0007] Based on the voxel coordinate information and the at least one facial attribute feature, a voxel feature is constructed;

[0008] The voxel features are converted into corresponding two-dimensional pixel features, and the target face image is rendered based on the pixel features.

[0009] Secondly, this application provides a model generation method, including:

[0010] Determine camera pose sample information, at least one facial attribute sample feature, and sample image;

[0011] The camera pose sample information and the at least one face attribute sample feature are input into the generator in the generative adversarial network. The generator constructs voxel sample coordinate information in three-dimensional space based on the camera pose sample information, constructs voxel sample features based on the voxel sample coordinate information and the at least one face attribute sample feature, converts the voxel sample features into pixel sample features in corresponding two-dimensional space, and renders the predicted image based on the pixel sample features.

[0012] The predicted image and the sample image are input into the discriminator in the generative adversarial network;

[0013] Based on the discrimination results of the discriminator, the model parameters of the generative adversarial network are adjusted.

[0014] Thirdly, this application provides an image processing method, including:

[0015] Determine the camera pose information and at least one target object attribute feature;

[0016] Using a generator in a generative adversarial network, voxel coordinate information in three-dimensional space is constructed based on the camera pose information;

[0017] Based on the voxel coordinate information and the at least one target object attribute feature, a voxel feature is constructed;

[0018] The voxel features are converted into corresponding two-dimensional pixel features, and a target image containing the target object is obtained by rendering based on the pixel features.

[0019] Fourthly, this application provides a model generation method, including:

[0020] Determine camera pose sample information, at least one target object attribute sample feature, and sample image;

[0021] The camera pose sample information and the at least one target object attribute sample feature are input into the generator in the generative adversarial network to construct voxel sample coordinate information in three-dimensional space based on the camera pose sample information, construct voxel sample features based on the voxel sample coordinate information and the at least one target object attribute sample feature, convert the voxel sample features into pixel sample features in corresponding two-dimensional space, and render a predicted image containing the target object based on the pixel sample features.

[0022] The predicted image and the sample image are input into the discriminator in the generative adversarial network to obtain the discrimination result;

[0023] Based on the discrimination results, the model parameters of the generative adversarial network are adjusted.

[0024] Fifthly, this application provides a computing device including a storage component and a processing component. The storage component stores one or more computer instructions. The one or more computer instructions are invoked and executed by the processing component to implement the image processing method as described in the first or third aspect above, or to implement the model generation method as described in the second or fourth aspect above.

[0025] This embodiment of the application is based on camera pose information and at least one facial attribute feature. Using a generator in a pre-trained generative adversarial network, voxel coordinate information in three-dimensional space can be constructed based on the camera pose information. Based on the voxel coordinate information and the at least one facial attribute feature, voxel features are constructed, and these voxel features are converted into corresponding pixel features in two-dimensional space. Based on these pixel features, a target face image can be rendered. In the face image generation process, by combining camera pose information and at least one facial attribute feature, voxel features in three-dimensional space can be constructed, introducing three-dimensional information. Then, the voxel features are converted into pixel features in two-dimensional space for rendering. Because more information is introduced during image generation, the image quality of the generated target face image can be improved, thereby increasing the accuracy of image generation.

[0026] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart of an embodiment of a model generation method provided in this application is shown;

[0029] Figure 2 A flowchart of one embodiment of an image processing method provided in this application is shown;

[0030] Figure 3 A flowchart of yet another embodiment of the model generation method provided in this application is shown;

[0031] Figure 4 This paper illustrates a schematic diagram of the structure of a generative adversarial network provided in an embodiment of this application in a practical application.

[0032] Figure 5 A flowchart of yet another embodiment of an image processing method provided in this application is shown;

[0033] Figure 6a This illustration shows an exemplary interactive scenario to which the technical solutions of this application are applicable;

[0034] Figure 6b This paper illustrates a comparative diagram of image generation effects in a practical application based on an embodiment of this application.

[0035] Figure 7 This paper shows a schematic diagram of the structure of an embodiment of a model generation apparatus provided in this application;

[0036] Figure 8 This application provides a schematic diagram illustrating the structure of one embodiment of a computing device.

[0037] Figure 9 A schematic diagram of one embodiment of an image processing apparatus provided in this application is shown. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0039] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0040] The technical solutions of this application can be applied to various application scenarios where there is a need for image generation, such as training image generation scenarios or face attribute editing scenarios, where face images with specific attributes are generated by changing certain attributes of the face.

[0041] In the process of developing this application, the inventors discovered that in image generation technologies, taking the generation of facial images as an example, generative adversarial networks (GANs) can be used. Based on facial attribute features extracted from real faces, the GAN is trained, enabling the generator within the GAN to generate corresponding facial images based on given facial attribute features. However, the image quality generated in this way is not good. Further research revealed that because faces are complex and diverse, possessing different facial structures, poses, and expressions, and real facial images are two-dimensional images from which the extracted facial attribute features are not accurate enough, the accuracy of image generation is ultimately affected.

[0042] Based on the above findings, in order to improve the accuracy of image generation, the inventors have creatively proposed the technical solution of this application. In the embodiments of this application, during the image generation process, by combining camera pose information and at least one target object attribute feature, voxel features in three-dimensional space can be constructed, thus introducing three-dimensional information into the voxel features. Then, the voxel features are converted into pixel features in two-dimensional space for rendering. By converting to three-dimensional space and then mapping back to two-dimensional space, more information can be introduced during the image generation process, improving the effect of the target object in the generated image, thereby improving the accuracy of image generation.

[0043] In practical applications, the target object can specifically refer to a human face. In one or more embodiments below, the technical solution of this application will be mainly described using a human face as an example.

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] Figure 1 A flowchart of one embodiment of a model generation method provided in this application is shown. The method may include the following steps:

[0046] 101: Determine camera pose sample information, at least one target object attribute sample feature, and sample image.

[0047] Camera pose sample information, at least one target object attribute sample feature, and sample images can be used as training samples to input a generative adversarial network (GAN). Training a GAN requires a large number of training samples, and the training process can be performed in batches. A GAN mainly consists of a generator and a discriminator. Of course, it can also include other network structures, which will be described in detail in the embodiments below.

[0048] Camera pose sample information can include extrinsic parameter sample information and intrinsic parameter sample information. Extrinsic parameter sample information can be generated based on the camera position and pointing direction in the world coordinate system, and can include rotation matrices and translation vectors. Intrinsic parameter sample information can include transformation matrices that can transform 3D camera coordinates to 2D homogeneous image coordinates.

[0049] In practical applications, the target object can refer to, for example, a human face, a human body, an animal face, an animal body, or other objects whose attributes can be transformed. Attributes can refer to, for example, the target object's shape, texture, expression, posture, etc.

[0050] Sample images can refer to images with real target objects, such as real human face images.

[0051] In order to train the generative adversarial network, camera pose sample information and at least one target object attribute sample feature can be randomly selected from a dataset conforming to a corresponding mathematical distribution. Therefore, optionally, the determination of the camera pose sample information and at least one target object attribute sample feature can include:

[0052] Camera pose sample information is randomly selected from a camera pose dataset that conforms to the corresponding mathematical distribution;

[0053] At least one target object attribute sample feature is randomly selected from the feature dataset corresponding to at least one object attribute type that conforms to the corresponding mathematical distribution.

[0054] The sample images can be randomly selected from an image dataset, which can consist of images containing real target objects.

[0055] Thus, camera pose sample information, at least one target object attribute sample feature, and sample image constitute the training sample.

[0056] The camera pose dataset can be generated based on a given range of values ​​according to a corresponding mathematical distribution. This range can be manually set to fit the specific context. Alternatively, camera pose information can be extracted from each real-world image of the target object from an image dataset, and the range of camera pose values ​​can be obtained through statistical analysis of multiple camera pose data. Based on this range, a camera pose dataset conforming to a corresponding mathematical distribution can be generated. This mathematical distribution could be, for example, a uniform distribution.

[0057] For any object attribute type, the feature dataset can be generated based on a given range of values ​​according to its corresponding mathematical distribution. This range can be manually set based on the actual scenario. Alternatively, attribute features can be extracted from each real target object image in an image dataset, and the range of attribute feature values ​​can be obtained through statistical analysis of multiple attribute features. Based on this range, a feature dataset conforming to the corresponding mathematical distribution can be generated. The mathematical distribution conforming to the feature training sets corresponding to different object attribute types can include uniform distribution or standard normal distribution. For example, facial shape features and facial texture features can conform to a standard normal distribution, while facial expression coefficient features, facial pose features, and eye movement direction features can conform to a uniform distribution.

[0058] 102: Input camera pose sample information and at least one target object attribute sample feature into the generator in the generative adversarial network to construct voxel sample coordinate information in three-dimensional space based on camera pose sample information, construct voxel sample features based on voxel sample coordinate information and at least one target object attribute sample feature, convert voxel sample features into corresponding two-dimensional pixel sample features, and render a predicted image containing the target object based on the pixel sample features.

[0059] The generator can predict and obtain a predicted image containing the target object by performing a series of operations based on camera pose sample information and at least one target object attribute sample feature.

[0060] Voxels, short for volume elements, are used in 3D imaging and can refer to the smallest unit in a 3D image.

[0061] The voxel sample coordinate information can include the voxel coordinate position and viewpoint information in three-dimensional space. The viewpoint information can be a vector formed by the tangent of the viewpoint decomposition angle from the camera position to a certain voxel. This viewpoint decomposition angle is the angle formed by the line connecting the center of the predicted image to a point perpendicular to the predicted image and at a distance equal to the camera's focal length, connecting this point to each pixel of the predicted image, and then to the plane containing the predicted image. The voxel coordinate position is the coordinate position in the three-dimensional world coordinate system, which can be obtained by transforming the extrinsic and intrinsic parameter matrices in the camera pose sample information in the image coordinate system, camera coordinate system, and world coordinate system, consistent with traditional implementations. This application will not elaborate further on this aspect.

[0062] The generator may include a feature construction module for constructing voxel sample features based on voxel sample coordinate information and at least one target object attribute sample feature. This feature construction module can be trained during the training process of the generative adversarial network, and can be implemented, for example, using a multilayer perceptron.

[0063] Voxel sample features can be mapped back to pixel sample features in two-dimensional space. The generator can include a feature recognition module that can generate a predicted image containing the target object based on the pixel sample features. This feature recognition module can be trained during the training process of the generative adversarial network. It can perform upsampling and convolution operations on the pixel sample features to achieve the rendering operation. In practical applications, this feature recognition module can be implemented using a convolutional neural network.

[0064] Of course, the generator may also include a feature extraction module for constructing voxel sample coordinate information in three-dimensional space based on camera pose sample information. This feature extraction module can calculate and encode the camera pose sample information to obtain voxel sample coordinate information.

[0065] To further improve the accuracy of image generation, the training samples may optionally include at least one background attribute sample feature, such as background shape feature, background texture feature, and / or background pose feature.

[0066] Therefore, constructing voxel sample features based on voxel sample coordinate information and at least one target object attribute sample feature can be achieved by mapping voxel sample coordinate information and at least one target object attribute sample feature to foreground voxel sample features; mapping voxel sample coordinate information and at least one background attribute sample feature to background voxel sample features; and fusing foreground voxel sample features and background voxel sample features into voxel sample features.

[0067] Therefore, the generator may specifically include a foreground feature construction module and a background feature module. In addition, it may include a feature fusion module for fusing foreground voxel sample features and background voxel sample features to obtain the final voxel sample features and converting the voxel sample features into pixel sample features.

[0068] 103: Input the predicted image and sample image into the discriminator in the generative adversarial network to obtain the discrimination result.

[0069] 104: Adjust the model parameters of the generative adversarial network based on the discrimination results.

[0070] The discrimination result indicates the degree of similarity between the predicted image and the sample image. Through repeated adversarial interactions between the discriminator and the generator, the model parameters of the generative adversarial network are continuously adjusted until the degree of similarity meets the training requirements, i.e., training is complete, and the generative adversarial network is obtained. This application does not limit the specific training method of the generative adversarial network.

[0071] After training, the Generative Adversarial Network (GAN) can be used to generate images. Training the GAN allows it to learn the decoupling of various attribute features, enabling it to individually edit each attribute and generate images of target objects with specific attribute features. Furthermore, the GAN can reconstruct voxel features in 3D space. By converting to 3D space and then mapping it back to 2D space, more information can be introduced during image generation, such as spatial effects in facial poses. Therefore, the technical solution of this application improves feature representation, enhances the representation of target objects in the generated images, and thus improves the accuracy of image generation.

[0072] like Figure 2 The diagram shown is a flowchart of one embodiment of an image processing method provided in this application. The method may include the following steps:

[0073] 201: Determine the camera pose information and at least one target object attribute feature.

[0074] The camera pose information can be randomly selected from a camera pose dataset that conforms to a corresponding mathematical distribution. The method for determining this camera pose dataset can be found in the previous embodiments.

[0075] The determination of at least one target object attribute feature can be implemented in various ways. For example, it can be combined with the actual application scenario, using fixed-configuration target object attribute features, user-set target object attribute features, and / or target object attribute features extracted from specific face images, etc., which will be described in detail in the corresponding embodiments below.

[0076] 202: Using generators in generative adversarial networks, voxel coordinate information in 3D space is constructed based on camera pose information.

[0077] By inputting the camera pose information and at least one target object attribute feature into the generator, the operations in steps 202 to 204 can be achieved.

[0078] The generator can first construct voxel coordinate information in 3D space based on camera pose information. Voxel coordinate information can include voxel coordinate position and viewpoint information. The viewpoint information can be a vector formed by the tangent of the viewpoint decomposition angle from the camera position to the voxel. This viewpoint decomposition angle is the angle formed by the line connecting the center of the predicted image to the point perpendicular to the predicted image and at a distance equal to the camera's focal length, connecting this point to each pixel of the predicted image, and then to the plane containing the predicted image. The voxel coordinate position is the coordinate position in the 3D world coordinate system, which can be obtained by transforming the extrinsic and intrinsic parameter matrices from the camera pose sample information in image coordinates, camera coordinates, and world coordinates, consistent with traditional implementations. This application will not elaborate further on this aspect.

[0079] 203: Construct voxel features based on voxel coordinate information and at least one target object attribute feature.

[0080] The generator may include a feature building module for constructing voxel features based on voxel coordinate information and at least one target object attribute feature. This feature building module can be trained during the generative adversarial network training process, and can be implemented, for example, using a multilayer perceptron.

[0081] 204: Convert voxel features into corresponding two-dimensional pixel features, and render the target image containing the target object based on the pixel features.

[0082] Voxel features can be mapped back to pixel features, achieving the goal of mapping from three-dimensional space to two-dimensional space. The generator can include a feature recognition module, which can generate a target image containing the target object based on pixel features. This feature recognition module can be trained during the training process of the generative adversarial network. It can perform upsampling and convolution operations on pixel features to achieve rendering operations. In practical applications, this feature recognition module can be implemented using a convolutional neural network.

[0083] Of course, the generator can also include a feature extraction module, which calculates and encodes the camera pose information to obtain voxel coordinate information.

[0084] To further improve the accuracy of image generation, optionally, at least one background attribute feature can be determined, thereby constructing a foreground voxel feature based on voxel coordinate information and at least one target object attribute feature; constructing a background voxel feature based on voxel coordinate information and at least one background attribute feature; and fusing the foreground voxel feature and the background voxel feature to obtain the voxel feature.

[0085] In this embodiment, by combining camera pose information and at least one target object attribute feature, voxel features can be constructed in three-dimensional space, thus introducing three-dimensional information into the voxel features. Then, the voxel features are converted into pixel features in two-dimensional space for rendering. By converting to three-dimensional space and then mapping back to two-dimensional space, more information can be introduced during the image generation process, improving the effect of the target object in the generated image, thereby improving the accuracy of image generation.

[0086] In practical applications, the target object can specifically refer to a human face. In one or more embodiments below, the technical solution of this application will be mainly described using a human face as an example.

[0087] like Figure 3 The diagram shown is a flowchart of another embodiment of a model generation method provided in this application. The method may include the following steps:

[0088] 301: Determine camera pose sample information, at least one facial attribute sample feature, and sample image.

[0089] Camera pose sample information, at least one target object attribute sample feature, and sample images can be used as training samples to input a generative adversarial network (GAN). Training a GAN requires a large number of training samples, and the training process can be performed in batches. A GAN mainly consists of a generator and a discriminator. Of course, it can also include other network structures, which will be described in detail in the embodiments below.

[0090] Camera pose sample information can include extrinsic parameter sample information and intrinsic parameter sample information. Extrinsic parameter sample information can be generated based on the camera position and pointing direction in the world coordinate system, and can include rotation matrices and translation vectors. Intrinsic parameter sample information can include transformation matrices that can transform 3D camera coordinates to 2D homogeneous image coordinates.

[0091] The at least one facial attribute sample feature may include the sample features of various attributes possessed by the face, such as facial shape sample features, facial texture sample features, facial pose sample features, facial expression coefficient sample features, and eye movement direction sample features, etc.

[0092] Among them, face shape determines face shape, face texture can determine skin color, face pose refers to head rotation posture information, and face expression coefficient refers to the weight value of each expression base when multiple expression bases are combined into one expression. An expression base can refer to a set of three-dimensional face models with the same topological result, which can represent the principal components of each face expression. Currently, it can be divided into 52 expression bases. The face expression coefficient can refer to the weight value of each of the 52 expression bases. Different face expression coefficients will eventually produce different face expressions.

[0093] The sample image can refer to a real human face image.

[0094] The camera pose sample information and at least one facial attribute sample feature are randomly selected from a dataset conforming to a corresponding mathematical distribution. Therefore, optionally, the determination of the camera pose sample information and at least one facial attribute sample feature may include:

[0095] Camera pose sample information is randomly selected from a camera pose dataset that conforms to the corresponding mathematical distribution;

[0096] At least one facial attribute sample feature is randomly selected from the feature dataset corresponding to at least one facial attribute type that conforms to the corresponding mathematical distribution.

[0097] The sample images can be randomly selected from the image dataset.

[0098] Thus, the camera pose sample information, at least one target object attribute sample feature, and the sample image constitute the training sample.

[0099] This camera pose dataset can be generated based on a given range of values ​​according to a corresponding mathematical distribution. This range can also be manually set to fit the specific context. Alternatively, camera pose information can be extracted from each real target image in an image dataset, and the range of camera pose values ​​can be obtained through statistical analysis of multiple camera pose data. Based on this range, a camera pose dataset conforming to a corresponding mathematical distribution can be generated. This mathematical distribution could be, for example, a uniform distribution.

[0100] The feature dataset corresponding to any type of facial attribute can be generated based on a given value range according to its corresponding mathematical distribution. This value range can also be manually set based on the actual context. Alternatively, facial attribute features can be extracted from each real face image in an image dataset, and the value range of multiple facial attribute features can be obtained through statistical analysis. Based on this value range, a feature dataset conforming to the corresponding mathematical distribution can be generated. The mathematical distribution conforming to the feature datasets corresponding to different facial attribute types can include uniform distribution or standard normal distribution. For example, facial shape features and facial texture features can conform to a standard normal distribution, while facial expression coefficient features, facial pose features, and eye movement direction features can conform to a uniform distribution.

[0101] Among these features, facial shape features, facial texture features, facial expression coefficient features, and eye movement direction features can be represented using latent codes (a type of feature vector), while facial pose features can be represented using a facial pose matrix. Camera pose sample information can also refer to the camera pose matrix, which consists of extrinsic and intrinsic parameter matrices.

[0102] 302: Input camera pose sample information and at least one face attribute sample feature into the generator in the generative adversarial network, so that the generator constructs voxel sample coordinate information in three-dimensional space based on the camera pose sample information, constructs voxel sample features based on the voxel sample coordinate information and at least one face attribute sample feature, converts the voxel sample features into pixel sample features, and renders the predicted image based on the pixel sample features.

[0103] Voxels, short for volume elements, are used in 3D imaging and can refer to the smallest unit in a 3D image.

[0104] The voxel sample coordinate information can include the voxel coordinate position and viewpoint information in three-dimensional space. The viewpoint information can be a vector formed by the tangent of the viewpoint decomposition angle from the camera position to a certain voxel. This viewpoint decomposition angle is the angle formed by the line connecting the center of the predicted image to a point perpendicular to the predicted image and at a distance equal to the camera's focal length, connecting this point to each pixel of the predicted image, and then to the plane containing the predicted image. The voxel coordinate position is the coordinate position in the three-dimensional world coordinate system, which can be obtained by transforming the extrinsic and intrinsic parameter matrices in the camera pose sample information in the image coordinate system, camera coordinate system, and world coordinate system, consistent with traditional implementations. This application will not elaborate further on this aspect.

[0105] The generator may include a feature construction module for constructing voxel sample features based on voxel sample coordinate information and at least one target object attribute sample feature. This feature construction module can be trained during the training process of the generative adversarial network, and can be implemented, for example, using a multilayer perceptron.

[0106] Voxel sample features can be mapped back to pixel sample features. The generator can include a feature recognition module that can generate a predicted image containing the target object based on the pixel sample features. This feature recognition module can be trained during the generative adversarial network training process. It can perform upsampling and convolution operations on the pixel sample features to achieve the rendering operation. In practical applications, this feature recognition module can be implemented using a convolutional neural network.

[0107] Of course, the generator may also include a feature extraction module for constructing voxel sample coordinate information in three-dimensional space based on camera pose sample information. This feature extraction module can calculate and encode the camera pose sample information to obtain voxel sample coordinate information.

[0108] To further improve image generation accuracy, the training samples may optionally include determining at least one background attribute sample feature. This at least one background attribute sample feature may include, for example, background shape features, background texture features, and / or background pose features. This at least one background attribute sample feature may be a fixed feature, and the background attribute sample features may be the same for each training iteration. Alternatively, each time the training cost is constructed, the features may be randomly selected from the feature training dataset corresponding to at least one background attribute type that conforms to the corresponding mathematical distribution.

[0109] Therefore, constructing voxel sample features based on voxel sample coordinate information and at least one face attribute sample feature can be achieved by mapping voxel sample coordinate information and at least one face attribute sample feature to foreground voxel sample features; mapping voxel sample coordinate information and at least one background attribute sample feature to background voxel sample features; and fusing foreground voxel sample features and background voxel sample features into voxel sample features.

[0110] Therefore, the generator may specifically include a foreground feature construction module and a background feature construction module, and may also include a feature fusion module for fusing foreground voxel sample features and background voxel sample features to obtain the final voxel sample features.

[0111] 303: Input the predicted image and sample image into the discriminator in the generative adversarial network.

[0112] 304: Adjust the model parameters of the generative adversarial network based on the discriminator's discrimination results.

[0113] The discrimination result indicates the degree of similarity between the predicted image and the sample image. Through repeated adversarial interactions between the discriminator and the generator, the model parameters of the generative adversarial network are continuously adjusted until the degree of similarity meets the training requirements, i.e., training is complete, and the generative adversarial network is obtained. This application does not limit the specific training method of the generative adversarial network.

[0114] In some embodiments, to further improve model accuracy and thus image generation accuracy, the method may further include:

[0115] The predicted image is input into the recognizer in the generative adversarial network, and the recognizer extracts one or more facial attribute prediction features from the predicted image.

[0116] Compare the predicted features of one or more attributes with the corresponding facial attribute sample features;

[0117] Based on the comparison results, the model parameters of the generative adversarial network are adjusted.

[0118] That is, the generative adversarial network may also include a recognizer connected to the generator output. The recognizer can extract at least one target face attribute prediction feature from the prediction map generated by the generator. The at least one target face attribute prediction feature can be determined according to actual needs. For example, if the purpose of training the generative adversarial network is to generate an image with a specific expression, then the at least one target face attribute may include the facial expression coefficient and the direction of eye movement. Or, if the purpose of training the generative adversarial network is to generate an image with a specific shape or texture, then the at least one target face attribute may include the face shape or face texture, etc.

[0119] By comparing the predicted features of facial attributes belonging to the same facial attribute type with the features of facial attribute samples, and based on the comparison results combined with loss functions and gradient backpropagation algorithms, the model parameters of the generative adversarial network can be adjusted until they meet the training requirements.

[0120] In other words, generative adversarial networks can combine the discrimination results of the discriminator and the recognition result to adjust the model parameters until a generative adversarial network that meets the training requirements is obtained.

[0121] This recognizer can be trained through the regenerative adversarial network training process, or it can be implemented using a pre-trained model that can extract the target face attribute features. In practical applications, this recognizer can be implemented using a regression model, for example.

[0122] The following is combined with Figure 4 The diagram shows an exemplary structure of a generative adversarial network, which explains the specific generation process of a generative adversarial network.

[0123] like Figure 4 As shown, a generative adversarial network may include, for example, a generator 100, a discriminator 200, and may also include a recognizer 300. The generator 100 may include a feature extraction module 401, a foreground feature construction module 402, a background feature construction module 403, a feature fusion module 404, and a feature recognition module 405.

[0124] The foreground feature construction module 402 and the background feature construction module 403 can be implemented using a multilayer perceptron. The special recognition module 405 can be implemented using a convolutional neural network.

[0125] The camera pose sample information is input into the feature extraction module 401. The feature extraction module 401 can calculate the voxel coordinate position x and the view information d from the camera pose sample information. It can also encode the voxel coordinate position x and the view information d respectively. The encoded voxel coordinate position x and the view information d constitute the voxel sample coordinate information in three-dimensional space.

[0126] At least one facial attribute sample feature and voxel sample coordinate information can be input into the foreground feature construction module 402, which then maps and obtains the foreground voxel sample feature f for each foreground voxel. ij F Here, j represents a certain line of sight, and i represents a voxel on line of sight j. Furthermore, the density value σ corresponding to the feature of each foreground voxel sample can be obtained. ij F The density value can represent the weight coefficient of the foreground voxel sample features.

[0127] At least one background attribute sample feature and voxel sample coordinate information can be input into the background feature construction module 403, so that the background voxel sample feature f of each background voxel can be obtained by mapping from the background feature construction module 403. ij B and density value σ ij B .

[0128] Subsequently, the feature fusion module 404 can first perform weighted fusion of foreground voxel sample features and background voxel sample features at the same voxel coordinate position according to their respective density values, thereby obtaining voxel sample features and their corresponding density values. One fusion method is, for example, to obtain the density value corresponding to a voxel sample feature by adding the density values ​​of the foreground and background voxel sample features at the same position. The weighting coefficient of the foreground voxel sample feature can be the ratio of its corresponding density value to the density sum, and the weighting coefficient of the background voxel sample feature can also be the ratio of its corresponding density value to the density sum. Then, the foreground and background voxel sample features are weighted and summed according to their respective weighting coefficients to obtain the voxel sample features. The specific fusion method can be implemented according to the following first fusion formula:

[0129]

[0130] Where C represents the voxel sample feature of a certain voxel in the fused voxel feature space. And the density value σ; m is equal to 1 or 2, f1 can represent the foreground voxel sample feature, σ1 can represent the density value of the foreground voxel sample feature, f2 can represent the background voxel sample feature, and σ2 can represent the density value of the background voxel sample feature.

[0131] Furthermore, the feature fusion module 404 can also convert voxel sample features into pixel sample features. For example, it can fuse the features of all voxel samples located under the same line of sight to obtain a single pixel sample feature. One fusion method is to obtain the feature through weighted summation, where the weighting coefficient of each voxel sample feature can be obtained based on the distance between its corresponding voxel and its neighboring voxels under the same line of sight. Specifically, this can be implemented according to the following second fusion formula:

[0132]

[0133] Where Ns represents the number of voxels located in the same line of sight; δ n f represents the distance between the nth voxel and its adjacent voxel under the same line of sight. n σ represents the voxel sample features of the nth voxel under the same line of sight. n τ represents the density value of the nth voxel under the same line of sight.n α n It can represent the weighting coefficients of the voxel sample features of the nth voxel.

[0134] The pixel sample features generated by the feature fusion module 404 can be input into the feature recognition module 405, which then renders the predicted image. This feature recognition module can, for example, be implemented using a convolutional neural network, performing upsampling and convolution operations on the features of each pixel sample to predict and render an image containing a face.

[0135] The predicted image rendered by the feature recognition module 405 can be input into the discriminator 200 and the recognizer 300. Similarly, the sample image, i.e., the real face image, is also input into the discriminator 200. The discriminator 200 determines the similarity between the sample image and the predicted image. The recognizer 300 extracts at least one target face attribute prediction feature from the predicted image, such as facial expression coefficient prediction features and eye movement direction prediction features, and compares them with the facial expression coefficient sample features and eye movement direction prediction features, respectively. Based on the comparison results of the recognizer 300 and the discrimination results of the discriminator 200, they can be used together to adjust the model parameters of the generative adversarial network (GAN), thereby training the GAN. Through repeated adversarial interactions between the discriminator, recognizer, and generator, the attribute decoupling capability of the generator is continuously strengthened.

[0136] Once the Generative Adversarial Network (GAN) has been trained, its generator can be used to generate images. By training the GAN, it can learn to decouple various attribute features, thereby enabling the trained GAN to edit each attribute individually and generate face images with specific facial attributes. Figure 5 This paper introduces the technical solution of this application from the perspective of image generation. Figure 5 The image processing method shown may include the following steps:

[0137] 501: Determine the camera pose information and at least one facial attribute feature.

[0138] 502: Using a generator in a generative adversarial network, voxel coordinates in 3D space are constructed based on camera pose information.

[0139] By inputting camera pose information and at least one facial attribute feature into the generator, the operations in steps 502 to 504 can be achieved.

[0140] 503: Construct voxel features based on voxel coordinate information and at least one facial attribute feature.

[0141] 504: Convert voxel features into corresponding two-dimensional pixel features, and render the target face image based on the pixel features.

[0142] Figure 5 The illustrated embodiments and Figure 2 The difference shown is that the target object specifically refers to a human face; for other detailed operations, please refer to [link / reference]. Figure 2 As described in the illustrated embodiment.

[0143] In some embodiments, the method may further include:

[0144] Identify at least one background attribute feature;

[0145] Constructing voxel features based on voxel coordinate information and at least one facial attribute feature may include: constructing foreground voxel features based on voxel coordinate information and at least one facial attribute feature; constructing background voxel features based on voxel coordinate information and at least one background attribute feature; and fusing the foreground voxel features and background voxel features to obtain voxel features.

[0146] As can be seen from the preceding description, such as Figure 4 In the generative adversarial network shown, the generator may include a feature extraction module, a foreground feature construction module, a background feature construction module, a feature fusion module, and a feature recognition module. Specifically, the feature extraction module constructs voxel coordinate information in three-dimensional space based on camera pose information; the foreground feature construction module constructs foreground voxel features based on voxel coordinate information and at least one facial attribute feature; the background feature construction module constructs background voxel features based on voxel coordinate information and at least one background attribute feature; then, the feature fusion module fuses the foreground voxel features and background voxel features to obtain voxel features, and converts the voxel features into pixel features in corresponding two-dimensional space; the feature recognition module renders the target face image based on the pixel features, and the target face image is the face image with the at least one facial attribute feature.

[0147] The process of fusing foreground and background voxel features to obtain voxel features can be specifically achieved by weighted fusing foreground and background voxel sample features at the same voxel coordinate position. For example, the weighted fusing method can be performed according to the first fusing formula mentioned above. First, the density values ​​of the foreground and background voxel features at the same voxel coordinate position are added together to obtain a density sum. The weighting coefficient of the foreground voxel feature can be the ratio of its corresponding density value to the density sum, and the weighting coefficient of the background voxel feature can also be the ratio of its corresponding density value to the density sum. Then, the foreground and background voxel features are weighted and summed according to their respective weighting coefficients to obtain the voxel sample feature. This density sum is used as the density value corresponding to the voxel sample feature.

[0148] Specifically, converting voxel features into corresponding two-dimensional pixel features can be achieved by fusing all voxel features located under the same line of sight to obtain a single pixel feature. One fusion method is to obtain the feature through weighted summation, where the weighting coefficient of each voxel feature can be obtained based on the distance between its corresponding voxel and its neighboring voxels under the same line of sight, and can be implemented according to the second fusion formula mentioned above.

[0149] The camera pose information can be randomly selected from a camera pose dataset that conforms to a corresponding mathematical distribution. The method for determining this camera pose dataset can be found in the previous embodiment.

[0150] The determination of at least one facial attribute feature can be implemented in various ways. As an optional method, the determination of at least one facial attribute feature may include:

[0151] Extract at least one facial attribute feature from at least one specific face image; wherein, a specific face image is used to extract one or more facial attribute features.

[0152] In this application, the at least one specific face image can be uploaded by the user. The image processing method provided in this embodiment can be executed by the server, and the at least one specific face image can be uploaded by the user to the server via the client. Alternatively, the image processing method provided in this embodiment can also be executed by the client. The client can acquire the at least one specific face image from the user's environment and / or from the local system based on a user request. The client can be a face attribute editing application, such as a face-changing application that can transform face attributes.

[0153] Users can specify which facial attribute features to extract from a specific face image, meaning that at least one facial attribute feature can be extracted from at least one specific face image according to user instructions.

[0154] The resulting target face attribute is an image that integrates the facial attribute features of different specific faces. Taking two specific face images as an example, suppose we extract facial shape features, facial texture features, and facial pose features from the first specific face image, and extract facial expression coefficients and eye movement direction features from the second specific face image. The resulting target face image will then possess the facial shape, facial texture, and facial pose of the first specific face image, and the facial expression and eye movement direction of the second specific face image. In practical applications, for example, the first specific face image could be a selfie taken by the user, and the second specific face image could be an image of a celebrity uploaded by the user. This allows us to use a celebrity's face image to drive the facial expression changes of our own face image, and so on.

[0155] As an alternative approach, the determination of at least one facial attribute feature may include:

[0156] From the feature datasets corresponding to at least one type of face attribute that respectively satisfy the corresponding mathematical distribution, randomly select face attribute features to obtain at least one face attribute feature.

[0157] By randomly selecting at least one facial attribute feature, the obtained target face image can be a virtual face image. This achieves the purpose of generating a virtual face image.

[0158] Virtual face images can be used as training samples for some face processing models. Since virtual face images are not real face images, they can protect personal privacy and improve the security of data processing.

[0159] As another alternative, the determination of at least one facial attribute feature may include:

[0160] Sensing the feature setting operation, determine at least one set facial attribute feature;

[0161] This feature setting operation can be user-triggered. It can involve the client sensing the user's setting operation and notifying the server, thereby determining the at least one facial attribute feature being set.

[0162] To facilitate user setup, the method may optionally include:

[0163] Output multiple face effect prompts, such as prompts for the user to see the corresponding face display effect, like a happy face, a sad face, an old person's face, a child's face, a thinner face, a fatter face, etc. The feature setting operation can specifically refer to the selection operation for the face effect prompt.

[0164] The sensing of feature setting operations, determining at least one set facial attribute feature, may include:

[0165] Sensing the selection operation triggered by multiple face effect prompts, determine the selected target face effect;

[0166] Determine at least one facial attribute feature corresponding to the target facial effect.

[0167] As another alternative, the determination of at least one facial attribute feature may include:

[0168] Extract at least one specific facial attribute feature from at least one specific facial image, and the at least one specific facial attribute feature and the at least one target facial attribute feature constitute at least one facial attribute feature.

[0169] The at least one target facial attribute feature can be obtained through pre-configuration.

[0170] That is, the at least one facial attribute feature can be partially extracted from a specific facial image and partially extracted using a fixed target facial attribute feature.

[0171] The at least one specific face image can be uploaded by the user. In practical applications, a user can upload a specific face image, which can be their own face image. The user can obtain this image by taking a picture of their face through a client or by uploading it from a local system. The at least one specific face attribute feature can include, for example, face shape features, face texture features, and face pose features. The at least one target face attribute feature can include, for example, face expression coefficient features and eye movement direction features, thereby enabling the transformation of the expression of a specific face in a specific face image. Of course, the at least one target face attribute feature can also include face shape features or face texture features, etc. Other face attribute features are extracted from the specific face image, thereby enabling changes in the shape or texture of a specific face in a specific face image.

[0172] Furthermore, the at least one specific facial attribute feature can be specifically extracted from a specific facial image, which can be any image frame from the target video. The target video can be, for example, obtained by user uploads. Optionally, the method may further include:

[0173] A second video is generated by stitching together the face images corresponding to multiple image frames in the target video.

[0174] A second video can be generated by stitching together the multiple image frames in the target video according to their chronological order.

[0175] This allows for the modification of corresponding facial attribute features in a target video based on at least one target facial attribute feature, thereby obtaining a second video with the target facial attribute features.

[0176] Of course, the at least one target facial attribute feature can be extracted from a facial image or obtained through user settings, etc.

[0177] As another alternative, the determination of at least one facial attribute feature may include:

[0178] Identify at least one facial attribute feature corresponding to an element in the target audio data or target text data.

[0179] Among them, an element can be the smallest unit that constitutes the target audio data or the target text data. For example, for target audio data, it can refer to an audio frame, and for target text data, it can be a single word, etc.

[0180] In other words, this application can also determine at least one facial attribute feature based on target audio data or target text data. The at least one facial attribute feature corresponding to an element can be preset, etc.

[0181] In practical applications, in some embodiments, the method may further include:

[0182] According to the arrangement order of multiple elements corresponding to the target audio data or target text, the face images corresponding to the multiple elements are merged to obtain the video image;

[0183] The video footage is associated with target audio data or target text data to obtain the first video.

[0184] The first video can present an audiovisual effect where a virtual face speaks based on the target audio data, and the target text data can be displayed as subtitles. This method can be used in scenarios such as news broadcasting or robotic customer service to enable virtual faces to deliver news or provide customer service.

[0185] In addition, there are multiple ways to determine at least one background attribute feature. For example, it can be extracted from feature datasets that conform to the corresponding mathematical distribution of at least one background attribute type; or it can be obtained by sensing user settings operations, etc.

[0186] In some embodiments, after rendering the target face image based on pixel features, the method may further include:

[0187] Based on the target face image, obtain the image rendering result;

[0188] Output the image rendering result.

[0189] Optionally, when the technical solution of this application is executed by the server, the output image rendering result can be that the server sends the image rendering result to the client, and the client displays the image rendering effect; or when the technical solution of this application is executed by the client, the client displays the image rendering result.

[0190] The image rendering result may include, for example, the target face image, or it may match the target face image with corresponding audio data and / or text data, and fuse the target face image with the audio data and / or text data to obtain the image rendering result; of course, it may also be based on the target face image, combined with audio data and / or text data to generate a rendered video as the image rendering result, etc.

[0191] Figure 6a The illustration shows an exemplary interactive scenario in which the technical solution applicable to the embodiments of this application can be applied. In this interactive scenario, a client 601 and a server 602 are included, wherein the server 602 can be deployed according to... Figure 3 The generative adversarial network obtained by training in the illustrated embodiment.

[0192] Client 601 can sense user operations. For example, it can determine a first specific image from the local system based on the user's response request. Alternatively, it can call the image acquisition control to acquire a face image based on the user's response request.

[0193] In addition, client 601 can also display multiple face effect prompts to guide the user to select the target face effect.

[0194] Client 601 can send a first specific image and the target face effect to server 602. Assume the target face effect corresponds to a specific facial expression, such as a laughing expression.

[0195] The server-side 602 can determine facial expression coefficients, eye movement direction features, etc., based on the target face effect. Then, it can extract facial texture features, facial pose features, and facial shape features from the first specific image.

[0196] Subsequently, server 602 can generate the target face image based on the facial expression coefficient features, eye movement direction features, and face shape features corresponding to the target face effect, as well as the facial texture features and facial pose features extracted from the first specific image, and at least one randomly selected background attribute feature, combined with randomly selected camera pose information, using the generator in the generative adversarial network.

[0197] The server 602 can send the target face image as an image rendering effect of the first specific face image to the client 601, and the client 601 can display the target face image.

[0198] Based on the user's actions, the client can choose to save or share the target face image, etc.

[0199] like Figure 6b The diagram shows a comparative display of a first specific image 600 transformed into a target face image 700.

[0200] Figure 6a as well as Figure 6b This is merely an example illustrating that the technical solution of this application can be applied to one use case. It is understood that the use cases of the technical solution of this application are not limited to this.

[0201] Figure 7 This application provides a schematic diagram of the structure of a model generation apparatus according to one embodiment. The apparatus may include:

[0202] The first determining module 701 is used to determine camera pose sample information, at least one target object attribute sample feature, and sample image;

[0203] The first prediction module 702 is used to input camera pose sample information and at least one target object attribute sample feature into the generator in the generative adversarial network, to construct voxel sample coordinate information in three-dimensional space based on the camera pose sample information, to construct voxel sample features based on the voxel sample coordinate information and at least one target object attribute sample feature, to convert the voxel sample features into pixel sample features, and to render a predicted image containing the target object based on the pixel sample features.

[0204] The first discrimination module 703 is used to input the predicted image and the sample image into the discriminator in the generative adversarial network to obtain the discrimination result;

[0205] The first training model, 704, is used to adjust the model parameters of the generative adversarial network based on the discrimination results.

[0206] In practical applications, the target object can specifically refer to a human face.

[0207] In some embodiments, the device may further include:

[0208] The first recognition module is used to input the predicted image into the recognizer in the generative adversarial network, and the recognizer extracts at least one target face attribute prediction feature from the predicted image; and compares the at least one target face attribute prediction feature with the corresponding face attribute sample feature.

[0209] This first training module can also be used to adjust the model parameters of the generative adversarial network based on the comparison results.

[0210] In some embodiments, the first determining module is further configured to determine at least one background attribute sample feature;

[0211] The first prediction module is also used to input at least one background attribute sample feature into the generator;

[0212] The first prediction module constructs voxel sample features based on voxel sample coordinate information and at least one face attribute sample feature. Specifically, it maps voxel sample coordinate information and at least one face attribute sample feature to foreground voxel sample features; maps voxel sample coordinate information and at least one background attribute sample feature to background voxel sample features; and fuses foreground voxel sample features and background voxel sample features into voxel sample features.

[0213] Figure 7 The model generation device described above can perform Figure 1 The implementation principle and technical effects of the model generation method described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the model generation apparatus in the above embodiments performs its operations have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0214] Furthermore, embodiments of this application also provide a computing device, such as... Figure 8 As shown, the computing device may include a storage component 801 and a processing component 802; the storage component 801 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component to achieve... Figure 1 The model generation method of the embodiment shown or Figure 2 The model generation method of the embodiment shown.

[0215] The processing component 802 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0216] Storage component 801 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0217] Of course, the computing device may also include other components, such as input / output interfaces, communication components, etc.

[0218] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc. Communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0219] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform. In this case, the computing device can refer to a cloud server, and the aforementioned processing components, storage components, etc., can be basic server resources rented or purchased from the cloud computing platform.

[0220] When the computing device is a server, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. It can be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0221] Furthermore, embodiments of this application also provide a computer-readable storage medium that, when storing a computer program, can achieve [the desired functionality] when executed by a computer. Figure 1 The model generation method of the embodiment shown or Figure 2 The model generation method of the illustrated embodiment. The computer-readable medium may be included in the computing device described in the above embodiments; or it may exist independently and not assembled into the computing device.

[0222] Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0223] Furthermore, embodiments of this application also provide a computer program product, which includes a computer program carried on a computer-readable storage medium, wherein the computer program, when executed by a computer, can perform the functions described above. Figure 1 Fire or Figure 3 The model generation method of the embodiment shown.

[0224] In such embodiments, the computer program may be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by the processing component, it performs the various functions defined in the system of this application.

[0225] Figure 9 This application provides a schematic diagram of the structure of an image processing apparatus according to one embodiment. The apparatus may include:

[0226] The second determining module 901 is used to determine camera pose information and at least one target object attribute feature;

[0227] The first construction module 902 uses a generator in a generative adversarial network to construct voxel coordinate information in three-dimensional space based on camera pose information;

[0228] The second construction module 903 constructs voxel features based on voxel coordinate information and at least one facial attribute feature;

[0229] The first rendering module 904 is used to convert voxel features into pixel features in the corresponding two-dimensional space, and to render a target image containing the target object based on the pixel features.

[0230] In practical applications, the target object can specifically refer to a human face, and the target image is the target human face image.

[0231] In some embodiments, the second determining module is further configured to determine at least one background attribute feature;

[0232] The second construction module can construct background voxel features based on voxel coordinate information and at least one background attribute feature; and fuse the foreground voxel features and background voxel features to obtain voxel features.

[0233] In some embodiments, the second determining module can be specifically used for:

[0234] Extract at least one facial attribute feature from at least one specific facial image; wherein, a specific facial image is used to extract one or more facial attribute features;

[0235] Alternatively, from the feature dataset corresponding to at least one type of face attribute that satisfies the corresponding mathematical distribution, randomly select face attribute features to obtain at least one face attribute feature.

[0236] Alternatively, it can sense the feature setting operation and determine at least one set facial attribute feature;

[0237] Alternatively, at least one specific facial attribute feature is extracted from at least one specific facial image, and at least one specific facial attribute feature and at least one target facial attribute feature constitute at least one facial attribute feature;

[0238] Alternatively, identify at least one facial attribute feature corresponding to an element in the target audio data or target text data.

[0239] In some embodiments, when at least one facial attribute feature is a facial attribute feature corresponding to an element in the target audio data or target text data, the apparatus may further include:

[0240] The first generation module is used to merge the face images corresponding to the multiple elements according to the arrangement order of the target audio data or target text to obtain a video frame; and associate the video frame with the target audio data or target text data to obtain a first video.

[0241] In some embodiments, where at least one facial attribute feature includes at least one specific facial attribute feature extracted from a specific facial image and at least one target facial attribute feature, the specific facial image is any image frame extracted from the target video.

[0242] The device may also include:

[0243] The second generation module stitches together the face images corresponding to multiple image frames in the target video to generate a second video.

[0244] In some embodiments, the device may further include:

[0245] The output module is used to obtain the image rendering result based on the target face image; and output the image rendering result.

[0246] In some embodiments, the second building module fuses the foreground voxel features and the background voxel features to obtain voxel features. Specifically, it fuses the foreground voxel features and the background voxel features corresponding to the same voxel coordinate position to obtain the voxel features corresponding to each voxel coordinate position.

[0247] The second building module converts voxel features into corresponding two-dimensional pixel features. Specifically, it can fuse all voxel features located in the same line of sight to obtain pixel features.

[0248] Figure 9 The model generation device described above can perform Figure 2 The implementation principle and technical effects of the image processing method described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the model generation device in the above embodiments performs its operations have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0249] Furthermore, embodiments of this application also provide a computing device that can be connected with... Figure 8 The computing devices shown may be the same or different computing devices. In this embodiment, the computing device may include a storage component and a processing component. The storage component stores one or more computer instructions, which are invoked and executed by the processing component to achieve... Figure 3 The image processing method of the embodiment shown or Figure 5 The image generation method of the illustrated embodiment. The specific structure of the computing device is similar to... Figure 8 The specific structure of the computing devices shown is the same, and will not be repeated here.

[0250] Furthermore, embodiments of this application also provide a computer-readable storage medium that, when storing a computer program, can achieve [the desired functionality] when executed by a computer. Figure 1 The model generation method of the embodiment shown or Figure 2 The model generation method of the illustrated embodiment. The computer-readable medium may be included in the computing device described in the above embodiments; or it may exist independently and not assembled into the computing device.

[0251] Furthermore, embodiments of this application also provide a computer program product, which includes a computer program carried on a computer-readable storage medium, wherein the computer program, when executed by a computer, can perform the functions described above. Figure 1 Fire or Figure 3 The model generation method of the embodiment shown.

[0252] In such embodiments, the computer program may be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by the processing component, it performs the various functions defined in the system of this application.

[0253] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0254] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized by, The method comprises: determining camera pose information and at least one facial attribute feature; constructing voxel coordinate information of the camera pose information in three-dimensional space by using a generator in a generative adversarial network; constructing voxel features based on the voxel coordinate information and the at least one facial attribute feature; converting the voxel features into pixel features corresponding to a two-dimensional space, and rendering a target face image based on the pixel features; The method further comprises: determining at least one background attribute feature; constructing voxel features based on the voxel coordinate information and the at least one facial attribute feature comprises: constructing foreground voxel features based on the voxel coordinate information and the at least one facial attribute feature; constructing background voxel features based on the voxel coordinate information and the at least one background attribute feature; fusing the foreground voxel features and the background voxel features to obtain voxel features.

2. The method of claim 1, wherein, The determination of the at least one facial attribute feature comprises: extracting at least one facial attribute feature from at least one specific face image; wherein one specific face image is used to extract one or more facial attribute features; or, randomly selecting facial attribute features from feature data sets corresponding to at least one facial attribute type that respectively satisfies a corresponding mathematical distribution to obtain at least one facial attribute feature; or, determining at least one facial attribute feature set according to a feature setting operation; or, extracting at least one specific facial attribute feature from at least one specific face image, and constructing the at least one facial attribute feature from the at least one specific facial attribute feature and at least one target facial attribute feature; or, determining at least one facial attribute feature corresponding to an element in target audio data or target text data.

3. The method of claim 2, wherein, In the case that the at least one facial attribute feature is at least one facial attribute feature corresponding to an element in the target audio data or the target text data, the method further comprises: merging face images corresponding to a plurality of elements in the target audio data or the target text according to the arrangement order of the plurality of elements to obtain a video frame; associating the video frame with the target audio data or the target text data to obtain a first video.

4. The method of claim 2, wherein, In the case that the at least one facial attribute feature comprises at least one specific facial attribute feature extracted from one specific face image and at least one target facial attribute feature, the specific face image is any image frame extracted from a target video; The method further comprises: splicing a second video from face images corresponding to a plurality of image frames of the target video.

5. The method of claim 1, wherein, After rendering a target face image based on the pixel features, the method further comprises: obtaining an image rendering result based on the target face image; outputting the image rendering result.

6. The method of claim 1, wherein, The fusion of the foreground voxel features and the background voxel features to obtain voxel features comprises: fusing foreground voxel features and background voxel features corresponding to the same voxel coordinate position to obtain voxel features corresponding to each voxel coordinate position; The converting the voxel feature into a pixel feature in a corresponding two-dimensional space comprises: Fusing all voxel features located in the same line of sight to obtain a pixel feature.

7. A model generation method characterized by comprising: Comprise: Determine camera pose sample information, at least one face attribute sample feature, and a sample image; Input the camera pose sample information and the at least one face attribute sample feature into a generator in a generative adversarial network to construct voxel coordinate information of the camera pose information in a three-dimensional space by the generator, construct a voxel sample feature based on the voxel sample coordinate information and the at least one face attribute sample feature, and convert the voxel sample feature into a pixel sample feature in a corresponding two-dimensional space, and render a predicted image based on the pixel sample feature; Input the predicted image and the sample image into a discriminator in the generative adversarial network; Adjust the model parameters of the generative adversarial network based on the discrimination result of the discriminator; The method further comprises: Determine at least one background attribute sample feature; Input the at least one background attribute sample feature into the generator; The constructing a voxel sample feature based on the voxel sample coordinate information and the at least one face attribute sample feature comprises: Map the voxel sample coordinate information and the at least one face attribute sample feature into a foreground voxel sample feature; Map the voxel sample coordinate information and the at least one background attribute sample feature into a background voxel sample feature; Fuse the foreground voxel sample feature and the background voxel sample feature into a voxel sample feature.

8. The method of claim 7, wherein, Further comprise: Input the predicted image into an identifier in the generative adversarial network to extract at least one target face attribute prediction feature from the predicted image by the identifier; Compare the at least one target face attribute prediction feature with a corresponding face attribute sample feature; Adjust the model parameters of the generative adversarial network based on the comparison result.

9. The method of claim 7, wherein, The determining camera pose sample information, at least one face attribute sample feature, and a sample image comprises: Randomly select camera pose sample information from a camera pose data set conforming to a corresponding mathematical distribution; Randomly select at least one face attribute sample feature from a feature data set corresponding to at least one face attribute type conforming to a corresponding mathematical distribution, respectively; Randomly select a sample image from an image data set conforming to a corresponding mathematical distribution; Constitute a training sample from the camera pose sample information, the at least one face attribute sample feature, and the sample image.

10. An image processing method characterized by, Comprise: Determine camera pose information and at least one target object attribute feature; Use a generator in a generative adversarial network to construct voxel coordinate information of the camera pose information in a three-dimensional space; Construct a voxel feature based on the voxel coordinate information and the at least one target object attribute feature; Convert the voxel feature into a pixel feature in a corresponding two-dimensional space, and render a target image containing a target object based on the pixel feature; The method further comprises: Determine at least one background attribute sample feature; Input the at least one background attribute sample feature into the generator; The constructing the voxel sample feature based on the voxel sample coordinate information and the at least one target object attribute feature comprises: mapping the voxel sample coordinate information and the at least one target object attribute feature into foreground voxel sample features; mapping the voxel sample coordinate information and the at least one background attribute sample feature into background voxel sample features; fusing the foreground voxel sample features and the background voxel sample features into voxel sample features.

11. A model generation method characterized by comprising: The method comprises: determining camera pose sample information, at least one target object attribute sample feature, and a sample image; inputting the camera pose sample information and the at least one target object attribute sample feature into a generator in a generative adversarial network to construct voxel coordinate information of the camera pose information in a three-dimensional space, constructing a voxel sample feature based on the voxel sample coordinate information and the at least one target object attribute sample feature, and converting the voxel sample feature into a pixel sample feature corresponding to a two-dimensional space, and rendering a predicted image containing a target object based on the pixel sample feature; inputting the predicted image and the sample image into a discriminator in the generative adversarial network to obtain a discrimination result; adjusting model parameters of the generative adversarial network based on the discrimination result; The method further comprises: determining at least one background attribute sample feature; inputting the at least one background attribute sample feature into the generator; The constructing the voxel sample feature based on the voxel sample coordinate information and the at least one target object attribute sample feature comprises: mapping the voxel sample coordinate information and the at least one target object attribute feature into foreground voxel sample features; mapping the voxel sample coordinate information and the at least one background attribute sample feature into background voxel sample features; fusing the foreground voxel sample features and the background voxel sample features into voxel sample features.

12. A computing device, comprising: The device comprises a storage component and a processing component, the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the image processing method of claim 1 or 10 or the model generation method of claim 7 or 11.

Images