Image generation method and device, computing equipment, storage medium and program product

Abstract

The invention provides an image generation method and apparatus, a computing device, a storage medium and a program product. The image generation method comprises the steps of obtaining a reference image and an initial image; performing object face analysis on the reference image to obtain a face attribute feature of a first object in the reference image, and performing feature extraction on the initial image based on a second object in the initial image to obtain an image feature; and based on the facial attribute features and the image features, generating a target image fusing the facial attributes of the first object. The technical effects that it can be ensured that the generated target image has the core facial features of the reference image uploaded by the user without a complex style migration process, the image is adaptive to the style of the initial image during generation, and meanwhile the facial attributes of the facial image uploaded by the user can be restored are achieved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image generation method. This application also relates to an image generation apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of image processing technology, personalized image customization has become an indispensable function, especially the "image face sculpting" that has become popular in recent years. This function allows users to upload a photo of themselves or a celebrity, and the system can automatically generate facial data that matches the photo. Users can adjust the specific parameters of the image to create a highly customized image that meets their needs.

[0003] However, current technologies generally focus on the specific shape of facial features, emphasizing the transfer of pixel-level features from real-life facial images to images of the target style, or directly combining pixel blocks with images of the target style. These methods, because they don't consider the compatibility between the target style image itself and the user-uploaded facial image, and because they overemphasize the specific shape of the face at the pixel level in the user-uploaded image, are prone to poor style transfer. This results in a final image that, even if facial features and other attributes are very close to the original user-uploaded image, still fails to achieve a perfect correspondence in facial shape, leading to a resemblance in form but not in spirit. Therefore, a method is urgently needed to solve this problem, enabling the final image generated by "image-based face sculpting" to both reproduce the user-uploaded facial image and adapt to the target style. Summary of the Invention

[0004] In view of this, embodiments of this application provide an image generation method to address the technical deficiencies existing in the prior art. Embodiments of this application also provide an image generation apparatus, a computing device, a computer-readable storage medium, and a computer program product.

[0005] According to a first aspect of the embodiments of this application, an image generation method is provided, comprising: Obtain the reference image and the initial image; Perform object facial analysis on the reference image to obtain the facial attribute features of the first object in the reference image, and extract features from the initial image based on the second object in the initial image to obtain image features; Based on the facial attribute features and the image features, a target image that fuses the facial attributes of the first object is generated.

[0006] According to a second aspect of the embodiments of this application, an image generation apparatus is provided, comprising: Acquisition module: configured to acquire reference and initial images; Extraction module: configured to perform object face analysis on the reference image to obtain facial attribute features of a first object in the reference image, and to extract features from the initial image based on a second object in the initial image to obtain image features; Fusion module: configured to generate a target image that fuses the facial attributes of the first object based on the facial attribute features and the image features.

[0007] According to a third aspect of the embodiments of this application, a computing device is provided, including: a memory and a processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the image generation method.

[0008] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores a computer program / instructions, which, when executed by a processor, implement an image generation method.

[0009] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement an image generation method.

[0010] The image generation method provided in this application extracts the facial attribute features of a user-uploaded reference image and the image features of an initial image, and then fuses the facial attribute features of the reference image with the image features of the initial image to generate the final target image. Instead of considering the pixel level, it focuses on more essential facial attribute features and integrates the facial attribute features of the reference image based on the image features of the initial image. It does not require a complex style transfer process and can ensure that the generated target image has the core facial features of the user-uploaded reference image. This solves the problem of adapting the style of the initial image while restoring the user-uploaded face image during image generation. Attached Figure Description

[0011] Figure 1 This is a flowchart of an image generation method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a method for creating a virtual game scene using a custom character model, as provided in one embodiment of this application. Figure 3 This is a schematic diagram of an image generation model provided in an embodiment of this application for an image generation method applied to a virtual game scene; Figure 4 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application; Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this application. Detailed Implementation

[0012] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0013] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first.

[0015] First, the terminology used in one or more embodiments of the present invention will be explained.

[0016] Unity Engine: A cross-platform game engine and development environment. It provides core tools and functions that simplify the creation, deployment, and operation of 2D and 3D games and interactive experiences (such as simulations and films), and is especially known for its ease of use and support for multi-platform publishing.

[0017] Maya (AutodeskMaya) engine: A professional-grade 3D computer graphics software primarily used for high-precision modeling, animation, simulation, and rendering.

[0018] ArcFace Loss Function: The ArcFace loss function is a widely used loss function in the field of face recognition. Its full name is Additive Angular Margin Loss. The purpose of ArcFace is to increase inter-class distance and reduce intra-class distance in the feature space, thereby improving the accuracy of face recognition. It is an improvement on the Softmax loss function, which enhances the discriminative power of features by adding a fixed margin in the angular space.

[0019] The Softmax function is a commonly used function in machine learning, widely used in the output layer of multi-class classification problems. It transforms a set of real numbers into a probability distribution, ensuring that the result is non-negative and sums to 1. Adaptive Normalization: Adaptive normalization is a technique used in deep learning to standardize features. It adjusts the mean and variance of the input data to improve its performance across different tasks and datasets. The main types of adaptive normalization include Adaptive Instance Normalization (AdaIN) and Adaptive Layer Normalization (adaLN). Adaptive Instance Normalization (AdaIN) is a technique for real-time arbitrary style transfer. It transfers style while preserving content by adjusting the mean and variance of the content image to match the statistics of the style image.

[0020] ResNet residual network: Residual networks are convolutional neural networks characterized by their ease of optimization and the ability to improve accuracy by increasing depth. The residual blocks inside use skip connections, which alleviates the gradient vanishing problem caused by increasing depth in deep neural networks.

[0021] GAN (Generative Adversarial Network): It consists of a generative model and a discriminative model. The generative model is responsible for capturing the distribution of sample data, while the discriminative model is generally a binary classifier that distinguishes between real data and generated samples. During training, one of them (the discriminative network or the generative network) is fixed, and the parameters of the other model are updated. This process is repeated iteratively, and eventually, the generative model can estimate the distribution of sample data. FaceNet network model: A face recognition model proposed by Google, designed to map face images onto feature vectors in Euclidean space to achieve face recognition. This model extracts features through a deep learning network and uses a triplet loss function for training to ensure that the facial feature distances of the same individual are small, while the facial feature distances of different individuals are large.

[0022] R-CNN (Region-based Convolutional Neural Networks) is an algorithm that applies deep learning to object detection. Proposed by Ross B. Girshick et al. in 2014, it was the first algorithm to successfully apply convolutional neural networks (CNNs) to object detection.

[0023] Faster R-CNN: Faster R-CNN is a further improvement on Fast R-CNN. Its main innovation is the introduction of a region proposal network, which replaces selective search.

[0024] DoG (Difference of Gaussians) is an image processing technique that performs edge detection and feature extraction by applying Gaussian filtering to an image and calculating its difference. It can be achieved by subtracting the Gaussian filtering results under different parameters.

[0025] This application provides an image generation method. This application also relates to an image generation apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0026] Figure 1 A flowchart of an image generation method according to an embodiment of this application is shown, which specifically includes the following steps: Step 102: Obtain the reference image and the initial image.

[0027] Reference Image: The reference image is the image input by the user. Specifically, the reference image needs to contain elements related to facial attributes. It can be a real photo, such as the user's ID photo, selfie, full-body photo, group photo, etc.; or it can be an image of other styles with elements related to facial attributes, such as screenshots from anime or video games, or portraits of virtual anime characters, etc.

[0028] In one implementation, facial attribute-related elements do not necessarily refer specifically to human face-related attribute elements. Any element that has facial-like attributes or a facial frame or outline can be considered as having facial attribute-related elements, including but not limited to animal faces in real scenes, mimicking robot faces, and facial images of characters such as fish-men and orcs in virtual scenes.

[0029] In one implementation, the elements related to facial attributes in the reference image do not necessarily have to have all the attribute elements of the entire face. Any element that conforms to facial attributes can be considered as having facial attributes, including but not limited to a photo of only half a face, a side profile photo, or even a feature of a facial organ such as a close-up of the eyes or a close-up of the nose.

[0030] The application can acquire reference and initial images, and the acquisition methods include, but are not limited to, uploading photos in the corresponding formats such as JPG and PNG; provide a camera capture function so that users can directly take pictures with their devices within the application and acquire photo data; or call up the user's historical data within the application and select suitable images for the user to upload from the historical data.

[0031] Initial image: An image with a specific style. The initial image can be a facial image containing specific filters, or with specific accessory elements, makeup elements, background elements, contrast, lighting intensity, etc. It can also be a digital image directly rendered by a specific virtual engine, such as a digital image rendered by a graphics rendering engine such as Unity, Unreal Engine, or Maya. The rendering engine in this embodiment includes, but is not limited to, the above-mentioned types. This application does not limit it here.

[0032] Optionally, the initial image can also be extracted from a scene with relevant elements, which is the target scene of the initial image. The target scene is a virtual scene with a certain style or a real scene with a certain filter. In one implementation, the target scene contains a specific style or filter. The user wants to generate a target image with the style corresponding to the target scene based on the facial attribute-related elements of the reference image. For example, generating a face photo with Chinese style elements from a face photo in an ID photo. Here, the target scene corresponds to parameters including but not limited to accessory elements, makeup elements, background elements, contrast, light intensity, color saturation, etc. It is a virtual scene corresponding to a digital image rendered by a specific virtual engine. The initial image extracted from the target scene is an image that conforms to the relevant attributes or style of the target scene.

[0033] In the implementation of this application, the relevant elements of the facial attributes of the reference image will be extracted and fused with the initial image to finally generate a target facial image that conforms to the relevant attributes or style of the initial image and has the relevant elements of the initial image. Obtaining the reference image and the initial image provides a data foundation for feature extraction, fusion, model training and other subsequent steps.

[0034] Step 104: Perform facial analysis on the reference image to obtain the facial attribute features of the first object in the reference image, and extract features from the initial image based on the second object in the initial image to obtain image features.

[0035] Performing object face analysis on a reference image can be done directly, or the facial region of the reference image can be determined first, and then object face analysis can be performed on the facial region of the reference image. After determining the facial region of the reference image, facial details can be adjusted first, and then object face analysis can be performed on the adjusted face.

[0036] Obtaining the facial attribute features of the first object in the reference image can be achieved by using a preset neural network model to extract features and obtain feature vectors, using manually designed operators or descriptors to capture feature information, or using a preset encoder to encode the image and obtain image features.

[0037] In one implementation, performing facial analysis on a reference image requires first identifying the facial regions of the reference image. Specific methods for identifying these regions include, but are not limited to, traditional feature-based methods, such as part-based models like DeformablePartModels, which model a face as a variable model composed of key parts (such as the left eye, right eye, nose, and mouth). These parts can deform within a certain range, and the face is located by detecting these parts and their geometric relationships. Alternatively, deep learning-based methods can be used, such as two-stage detectors like R-CNN or FASTR-CNN, which first generate a large number of candidate regions on the reference image, then classify or regress each candidate region to determine the facial regions of the reference image. In one implementation, if there are multiple facial regions in the reference image, the different facial regions can be separated and named as facial region 1, facial region 2, etc., and they can be treated as separate items for image generation. Different target images can be generated based on the initial image to provide to the user.

[0038] In one implementation, after determining the facial region of the reference image, further processing of the facial region is required, namely, accurately segmenting the facial region from unrelated parts such as the background, hair, and clothing. An alternative method is to crop out the pure facial region based on the detected face bounding box or the corrected / segmented region. The cropped image is then scaled to a uniform size (e.g., 112x112, 224x224) to facilitate subsequent feature extraction.

[0039] In one implementation, if the reference image is significantly affected by lighting, an algorithm, such as DoG (Difference of Gaussians), needs to be applied to the reference image to reduce the impact of lighting changes.

[0040] In one implementation, the reference image is processed to obtain the facial region of the reference image, and then the facial attribute features of the first object in the reference image are obtained based on the facial region of the reference image. The method of obtaining the features is to use a preset neural network model for feature extraction.

[0041] In one implementation, the reference image is directly input into a preset feature extraction model to extract features, thereby obtaining the facial attribute features of the first object in the reference image.

[0042] In one implementation, a neural network model trained by the ArcFace(AdditiveAngularMarginLoss, ArcFaceLoss) function is used to extract features from the facial region of the reference image or the processed reference image. ArcFace is a loss function used in deep learning, primarily for tasks such as face recognition and image retrieval. Its core purpose is to enhance the discriminative power of learned features, meaning that features of similar samples should be as close as possible in the embedding space, while features of different samples should be as far apart as possible. The ArcFace loss function achieves this by introducing angular margin, adding a parameter to the target angle for calculating the cosine value. This prevents ArcFace from directly calculating the loss using the cosine of the target angle. During model training, the angle between the feature vector and the target weights can be smaller, significantly improving the accuracy of face verification and recognition. It has achieved extremely high feature discrimination capabilities in deep face recognition tasks. The feature extraction model trained using the ArcFace function can extract the core features of the first object in the reference image without overly relying on the original pixels of the image. This decoupling from pixels to a certain extent ensures that the facial attribute features of the first object are the core features representing the facial attributes of the first object. This allows for better reconstruction of the facial attributes of the first object in subsequent feature fusion and image generation, making the generated target image closely resemble the first object.

[0043] In one implementation, the FaceNet neural network model is used to extract features from the facial regions of a reference image or a processed reference image. The efficient and compact embedding features learned using the FaceNet model effectively capture the global spatial structure information of the first object's face without overly relying on the original pixel details of the image. This highly generalizable embedding representation can largely decouple from specific pixel textures. These features, in subsequent feature fusion and image generation processes, help to more accurately reconstruct the overall facial structure and similarity of the first object, ensuring that the generated target image not only accurately reproduces the first object in key attributes.

[0044] Feature extraction is performed on the initial image based on the second object in the initial image to obtain image features. The second object in the initial image is a facial region in the initial image that has the style of the initial image. The specific extraction method can be to use a preset neural network model to extract features and obtain feature vectors, use manually designed operators or descriptors to capture feature information, or use a preset encoder to encode the image to obtain image features.

[0045] In one implementation, a preset encoder is used to encode a second object in the initial image to obtain the image features of the second object in the initial image.

[0046] In one implementation, feature extraction is performed on the initial image based on the second object in the initial image to obtain image features. Specifically, the extraction method may be to use a deep learning model framework to extract features from the initial image to obtain the feature vector of the initial image, which is the image feature of the second object in the initial image.

[0047] Step 106: Generate a target image that fuses the facial attributes of the first object based on the facial attribute features and the image features.

[0048] Facial attribute features and image features can be fused to obtain image fusion features. Then, a target image of the fused facial attributes of the first object can be generated based on the image fusion features. The generation method can be to input the fusion features into a preset decoder for decoding, or to use a deep generation model to perform the inverse process of feature extraction of the fusion features, and to inject the fusion features as conditional information into each step of the inverse process, and finally generate the target image of the fused facial attributes of the first object.

[0049] The above technical solution extracts the facial attribute features of the user-uploaded reference image and the image features of the initial image, and then fuses the facial attribute features of the reference image with the image features of the initial image to generate the final target image. It does not consider the pixel level but focuses on the more essential facial attribute features. It uses the image features of the initial image as a base and integrates the facial attribute features of the reference image. It does not require a complex style transfer process and can ensure that the generated target image has the core facial features of the user-uploaded reference image. It solves the problem of adapting the style of the initial image while restoring the user-uploaded face image during image generation. However, in order to facilitate feature extraction and image generation, and to make the effect of the final generated target image better, this application constructs a network model to achieve this effect, which will be described in detail in the following embodiments.

[0050] In one embodiment of this application, step 104 above, which involves performing object face analysis on the reference image to obtain the facial attribute features of the first object in the reference image, can be implemented through the following steps: Based on the reference image, the first feature processing unit of the image generation model is invoked to obtain the facial attribute features of the first object in the reference image.

[0051] In one implementation, the first feature processing unit may include a neural network model trained with FaceNet (which learns global facial embedding features based on a triplet loss function) to extract features from the facial regions of a reference image or a processed reference image. The efficient and compact embedding features learned using the FaceNet model effectively capture the global spatial structure information of the first object's face without overly relying on the original pixel details of the image. This highly generalizable embedding representation can largely decouple from specific pixel textures. These features, in subsequent feature fusion and image generation processes, help to more accurately reconstruct the overall facial structure and similarity of the first object, ensuring that the generated target image not only accurately reproduces the first object in key attributes.

[0052] In one implementation, the first feature processing unit may include a neural network model trained by the ArcFace (AdditiveAngularMarginLoss, ArcFaceLoss) function to extract features from the facial region of the reference image or the processed reference image. The feature extraction model trained by the ArcFace function can extract the core features of the first object in the reference image without overly relying on the original pixels of the image. It can achieve decoupling from the pixels to a certain extent, so that the facial attribute features of the first object are the core features representing the facial attributes of the first object. This can better restore the facial attributes of the first object in subsequent feature fusion and image generation, making the generated target image resemble the first object.

[0053] Accordingly, in step 104 above, feature extraction is performed on the initial image based on the second object in the initial image to obtain image features. This can be achieved through the following steps: Based on the initial image, the second feature processing unit of the image generation model is invoked to obtain the image features of the initial image.

[0054] For example, the second feature processing unit may be a preset neural network model, or contain manually designed operators or descriptors, or contain a preset encoder.

[0055] In one implementation, the second feature processing unit includes a preset encoder. The process of calling the second feature processing unit based on the initial image to obtain the image features of the initial image is to encode the initial image using the preset encoder included in the second feature processing unit to obtain the image features of the second object in the initial image.

[0056] In one implementation, the second feature processing unit is a preset neural network model. Based on the initial image, the second feature processing unit is called to extract features from the initial image based on the second object in the initial image. The process of obtaining image features is to use the preset neural network model to extract features from the initial image and obtain the feature vector of the initial image, which is the image feature of the second object in the initial image.

[0057] Accordingly, in step 106 above, a target image fused with the facial attributes of the first object is generated based on the facial attribute features and the image features. This can be achieved through the following steps: Based on facial attribute features and the image features, the feature fusion unit of the image generation model is invoked to obtain fused features.

[0058] The feature fusion unit of an image generation model can contain a pre-defined decoder.

[0059] In one implementation, the feature fusion unit includes a preset decoder. Based on facial attribute features and the image features, it calls the feature fusion unit of the image generation model to obtain fused features. The process involves performing feature fusion processing on facial attribute features and image features to obtain image fused features.

[0060] Based on the fusion features, the generation unit of the image generation model is invoked to generate a target image that fuses the facial attributes of the first object.

[0061] The generation unit of the image generation model can be composed of a preset depth generation model, or the generation unit can contain a preset decoder.

[0062] In one implementation, the generation unit of the image generation model is a preset depth generation model. Based on the fusion features, the process of calling the generation unit of the image generation model to generate a target image that fuses the facial attributes of the first object is to use a feature fusion unit composed of a preset depth generation model to perform the inverse operation of feature extraction on the fusion features, inject the fusion features as conditional information into each step of the inverse process, and finally generate a target image that fuses the facial attributes of the first object.

[0063] In one implementation, the generation unit of the image generation model includes a preset decoder. The process of generating a target image that fuses the facial attributes of the first object is to use the generation unit containing the preset decoder to decode the fusion features and obtain the target image that fuses the facial attributes of the first object.

[0064] Based on this, a pre-trained deep learning model containing a first feature processing unit, a second feature processing unit, a feature fusion unit, and a generation unit is constructed. This model can facilitate feature extraction, fusion, and target image generation. Furthermore, the model can be pre-trained to improve the quality of the target images generated by the model in specific applications.

[0065] In the above image generation process, when the first feature processing unit extracts features from the first object of the reference image to obtain the facial features of the first object of the reference image, since this application does not impose any restrictions on the reference image, and in order to meet the needs of users as much as possible, it is necessary to not restrict the category and source of the reference image as much as possible. This will lead to the first feature processing unit having poor robustness and poor feature adaptability when processing a wide variety of reference images, and a technical solution is urgently needed to solve this problem.

[0066] To address the aforementioned issues, this application provides an implementation in which the first feature processing unit may include a feature extraction layer and a feature transformation layer. Accordingly, the step of obtaining the facial attribute features of the first object in the reference image by calling the first feature processing unit of the image generation model based on the reference image can be specifically implemented as follows: Based on the reference image, the feature extraction layer is invoked to obtain the facial feature vector of the first object in the reference image; Based on the facial feature vector, a feature transformation layer is invoked to convert the facial feature vector into the facial attribute features of the first object.

[0067] In one implementation, the feature extraction layer can be composed of a preset neural network model, including but not limited to a neural network model trained by the ArcFace loss function, the FaceNet network model, etc. The first object of the reference image is input into the first feature processing unit, and features are extracted first through the feature extraction layer. Then, the extracted features are transformed by the feature transformation layer to obtain the facial attribute features of the first object of the reference image after feature adaptive transformation.

[0068] In one implementation, the feature transformation layer can be a two-layer fully connected layer. The two-layer fully connected layer can decouple the features in layers, strengthen the discriminative feature components related to facial attributes and identity, and suppress irrelevant variables. After feature extraction by the feature extraction layer, the two-layer fully connected layer can process the features, which can significantly improve the robustness of the model to imaging defects, environmental variations and adversarial attacks.

[0069] Based on this, by setting the first feature extraction unit to be constructed from a feature extraction layer and a feature transformation layer, the first object of the reference image can undergo adaptive feature transformation after the feature extraction layer extracts features, which makes the output facial attribute features more robust and enables the model to better cope with different types and styles of reference images.

[0070] In the image generation method of this application, during the process of obtaining image features of the initial image by calling the second feature processing unit of the image generation model based on the initial image, directly extracting image features from the initial image often results in semantic ambiguity of the features of the second object in the initial image and the lack of discriminative elements of facial elements. To address this problem, in one implementation provided by this application, the image generation model may further include a convolutional layer. Accordingly, the step of obtaining image features of the initial image by calling the second feature processing unit of the image generation model based on the initial image can be implemented in the following way: Based on the initial image, a convolutional layer is invoked to obtain the initial image features with a preset number of channels; Based on the initial image features with a preset number of channels, the second feature processing unit is invoked to obtain the image features of the initial image.

[0071] A convolutional layer is a type of neural network layer whose core operation is convolution. It uses a set of learnable convolutional kernels to slide across the initial input image with a certain stride. At each step, the convolutional kernel performs a dot product operation with the object elements in the area covered by the input image, and finally, an activation function is used to produce an output value at that position.

[0072] In one implementation, 64 convolutional kernels with a size of 3×3, a stride of 2, and padding of 1 can be used to detect the existence of local features such as edges, lines, corners, and specific textures in the input initial image. This results in an initial image feature with 64 channels, halved in length and width, which can preserve the integrity of the original spatial structure and extract machine-understandable semantic units, thus providing a good data foundation for subsequent feature extraction.

[0073] For example, the second feature processing unit may include a preset encoder.

[0074] In one implementation, the second feature processing unit includes a preset encoder. The process of calling the second feature processing unit based on the initial image to obtain the image features of the initial image is to use the preset encoder included in the second feature processing unit to encode the second object in the initial image to obtain the image features of the second object in the initial image.

[0075] Based on this, by setting up a convolutional layer, the initial image features after processing by the convolutional layer are input into the second feature processing unit, which makes the semantics of the image features of the initial image obtained by the second feature processing unit clearer and more suitable for feature fusion or feature injection.

[0076] In the image generation process of this application, since the image features of the initial image and the facial attribute features of the reference image belong to heterogeneous data spaces, their features cannot be effectively aligned during feature fusion, resulting in distorted facial structures or mixed styles in the generated image. To address this problem, in one implementation provided by this application, the image generation model may further include a residual network unit. Accordingly, after the step of calling the second feature processing unit of the image generation model based on the initial image to obtain the image features of the initial image, the following steps may also be included: Based on facial attribute features and image features, residual network units are invoked to perform feature fusion and obtain initial fused features.

[0077] In one implementation, a deep neural network model based on Residual Network Units (RNUs) is used to fuse facial attribute features and image features. The deep model structure formed by stacking RNUs can effectively learn the hierarchical representation of images, and is particularly adept at solving the gradient vanishing problem in deep network training, promoting the stable transmission of deep information. Through the design of cross-layer residual connections, the model's ability to reuse original input information and abstract features at all levels during forward propagation is enhanced. This structure allows the learned deep facial features to contain higher-level semantic information (such as identity and attributes) while also better preserving the low-level details of the input data, rather than relying solely on shallow mapping of the image's original pixel values. The features fused by the residual network are highly representative at all levels of abstraction, capable of capturing facial structure and detail changes from local to global perspectives, thus achieving a more robust representation. These multi-layered and stable deep features help to more comprehensively and accurately reconstruct the key attributes and detailed textures of the first object's face, making the subsequently generated target image closer to the visual features of the first object in terms of clarity and realism, and more closely resemble the first object.

[0078] In one implementation, the residual network unit can be a deep residual network model incorporating adaptive instance normalization to fuse facial attribute features and image features. This structure dynamically modulates the intermediate feature distribution of the residual network through the adaptive instance normalization module, injecting semantic information, including style, expression, and facial features, into the feature space. The deep residual network provides multi-level feature extraction capabilities, dynamically injecting facial attribute features from the reference image into the image features of the initial image to obtain initial fused features. Residual connections ensure gradient propagation stability and avoid mesh degradation issues.

[0079] Accordingly, the step of invoking residual network units for feature fusion based on facial attribute features and image features can be implemented in the following way: Based on facial attribute features and initial fusion features, the feature fusion unit of the image generation model is invoked to obtain fusion features.

[0080] In one implementation, the feature fusion unit includes a preset decoder. Based on facial attribute features and initial fusion features, it calls the feature fusion unit of the image generation model to obtain fusion features. The process of obtaining fusion features involves performing feature fusion processing on facial attribute features and initial fusion features to obtain image fusion features.

[0081] Based on this, in the process of obtaining fused features by calling the feature fusion unit of the image generation model, by constructing a residual network unit, the facial attribute features of the reference image and the image features of the initial image are first fused to obtain the initial fused features. Then, based on the initial fused features, facial attribute features and the image features of the initial image, feature fusion is performed, which can effectively align the image features of the initial image and the facial attribute features of the reference image, providing the necessary stability, dynamic adaptation capability and training guarantee for the transfer of facial attribute features to the image features of the initial image.

[0082] In the method of fusing initial fused features, facial attribute features, and image features of the initial image by calling the feature fusion unit of the image generation model to obtain fused features, the target image generated by the image generation unit after fusing the facial attributes of the reference image still suffers from problems such as local distortion, attenuation of expression features, and insufficient style consistency. To address this issue, in one implementation provided in this application, the second feature processing unit may further include multiple encoding layers. Accordingly, the step of calling the second feature processing unit of the image generation model based on the initial image to obtain the image features of the initial image can be specifically implemented as follows: Based on the initial image, multiple encoding layers are invoked to sequentially obtain multi-layer image features of the initial image.

[0083] For example, the purpose of setting multiple coding layers in the second feature processing unit is to perform downsampling processing on the feature map. The core objective is to compress the spatial dimensions including image height and width, while increasing the number of channels of the initial image features to extract more abstract high-dimensional features.

[0084] The multiple coding layers of the second feature processing unit can perform downsampling operations on the initial image features with a preset number of channels obtained after processing by the convolutional layer to achieve spatial compression, resulting in multi-layer image features of the initial image with lower resolution and a multiplied number of channels.

[0085] For example, the downsampling operation can be performed by sliding a convolutional kernel of a preset size across the image features with a preset stride. Each time the kernel moves, it skips some pixels, halving the size of the output feature map. The number of channels in the output feature map can be doubled by increasing the number of convolutional kernels to retain more high-order feature information. For example, if the initial image features are 256×256×64, after downsampling, the output image features are in the format of 8×8×1024.

[0086] In one implementation, the downsampling operation is performed multiple times, and the image features at the corresponding level obtained from each downsampling are recorded. For example, the obtained multi-layer image features can be used for subsequent feature fusion. Based on this, during the feature fusion process, the facial attribute features of the reference image and the initial fused features are fused together. The downsampling features at each resolution level carry the corresponding style of the initial image. Through step-by-step fusion, the microstructure of the output image can be maintained, ensuring style consistency.

[0087] Similarly, to better integrate the facial attribute features of the reference image into the image features of the initial image, and further improve the resemblance between the final generated target image and the reference image, in one implementation provided in this application, the feature fusion unit may also include multiple decoding layers. Accordingly, the step of calling the feature fusion unit of the image generation model based on facial attribute features and image features to obtain fused features can be implemented in the following way: Based on facial attribute features and multi-layer image features, multiple decoding layers are invoked to sequentially perform feature fusion decoding to obtain fused features.

[0088] For example, the purpose of setting multiple decoding layers in the feature fusion unit is to perform upsampling operations on the fused image features, which is the inverse process of the downsampling operation of the above-mentioned coding layer, so as to achieve spatial expansion, restore the height and width of the image features, and at the same time reduce the number of channels to gradually reconstruct details and accurate positioning information.

[0089] For example, upsampling can be performed on the feature map using transposed convolution with specific padding and stride settings. Taking transposed convolution as an example, it uses a convolution kernel of a preset size and a preset stride, and performs zero padding or sparse sampling as it slides across the feature map, doubling the spatial size of the output. Simultaneously, by reducing the number of convolution kernels used in this operation, the number of channels in the output feature map is halved. This allows abstract semantic features to be gradually transformed into detailed features that are closer to the original image. For example, a high-level feature input of 8×8×1024 can be output as a fine-grained image feature of 256×256×64 format after step-by-step upsampling processing by the decoding layer.

[0090] In one implementation, the upsampling operation is performed multiple times. The image features of the corresponding level obtained from each upsampling, the initial image features of the corresponding resolution level downsampled, and the facial attribute features of the reference image are fused to obtain new fused features. Then, the above upsampling operation and fusion process are repeated on the new fused features until the preset number of upsampling operations is completed.

[0091] Based on this, multiple decoding layers are constructed to upsample the fusion features. During the upsampling process, the facial attribute features of the reference image and the downsampled features of the initial image at the corresponding resolution level are fused together. This allows for a better fusion of the style attributes of the initial image (such as lighting, color, and texture style) and the facial attributes of the reference image. At the same time, the local detail features (such as edge structure and facial contours) of the downsampled layer are used for geometric correction, thereby achieving accurate transfer of style features while maintaining the structural accuracy of the generated image, and finally generating a high-quality target image.

[0092] The above describes the application process of the image generation model of this application. In order to ensure the quality of the target image generated by the image generation model, this application also provides a training method for the image generation model before applying it. The specific steps include six steps: The first step is to obtain a sample set, which includes sample reference images and sample scene images; For example, the source of the sample reference image can simulate the source of the reference image during the model application process. It can be widely obtained from real face photos on the network. The sample scene image corresponds to the initial image used during the model application process. The acquisition of the sample scene image can be using the default face shape of the sample scene or the face image under the style corresponding to the sample scene.

[0093] The second step is to call the first feature processing unit based on the sample reference image to obtain the sample facial attribute features of the first sample object in the sample reference image. In one implementation, the first feature processing unit may include a neural network model trained with FaceNet (which learns global facial embedding features based on a triplet loss function) to extract features from the facial regions of the sample reference image or the processed sample reference image. The efficient and compact embedding features learned using the FaceNet model can effectively capture the global spatial structure information of the first object's face without overly relying on the original pixel details of the image. This highly generalizable embedding representation can largely decouple from specific pixel textures. These features, in subsequent feature fusion and image generation processes, help to more accurately reconstruct the overall facial structure and similarity of the first object, enabling the generated predicted image to not only accurately reproduce the first object in key attributes.

[0094] In one implementation, the first feature processing unit may include a neural network model trained by the ArcFace (AdditiveAngularMarginLoss, ArcFaceLoss) function to extract features from the facial region of the sample reference image or the processed sample reference image. The feature extraction model trained by the ArcFace function can extract the core features of the first object in the sample reference image without overly relying on the original pixels of the image. It can achieve decoupling from pixels to a certain extent, so that the facial attribute features of the first object are the core features representing the facial attributes of the first object. This can better restore the facial attributes of the first object in subsequent feature fusion and image generation, making the generated target image resemble the first object.

[0095] The third step is to call the second feature processing unit based on the sample scene image to obtain the sample image features of the sample scene image; In one implementation, the second feature processing unit includes a preset encoder. The process of calling the second feature processing unit to obtain the image features of the sample scene image based on the sample scene image is to use the preset encoder included in the second feature processing unit to encode the sample scene image to obtain the image features of the second object in the sample scene image.

[0096] In one implementation, the second feature processing unit is a preset neural network model. Based on the sample scene image, the second feature processing unit is called to extract features from the initial image based on the second object in the sample scene image. The process of obtaining image features is to use the preset neural network model to extract features from the sample scene image to obtain the feature vector of the sample scene image, which is the image feature of the second object in the sample scene image.

[0097] Fourth step: Based on the facial attribute features of the sample and the image features of the sample, call the feature fusion unit to obtain the sample fusion features; The feature fusion unit of an image generation model can contain a pre-defined decoder.

[0098] In one implementation, the feature fusion unit includes a preset decoder. Based on facial attribute features and the image features, it calls the feature fusion unit of the image generation model to obtain fused features. The process involves performing feature fusion processing on facial attribute features and image features to obtain image fused features.

[0099] Fifth step: Based on the sample fusion features, call the generation unit to generate a predicted image that fuses the facial attributes of the first sample object; The generation unit of the image generation model can be composed of a preset depth generation model, or the generation unit can contain a preset decoder.

[0100] In one implementation, the generation unit of the image generation model is a preset deep generation model. Based on the fusion features, the process of calling the generation unit of the image generation model to generate a target image that fuses the facial attributes of the first object is to use a feature fusion unit composed of a preset deep generation model to perform the inverse operation of feature extraction on the fusion features, inject the fusion features as conditional information into each step of the inverse process, and finally generate a predicted image of the fused facial attributes of the first object.

[0101] In one implementation, the generation unit of the image generation model includes a preset decoder. The process of generating a target image that fuses the facial attributes of a first object is to use the generation unit containing the preset decoder to decode the fused features and obtain a predicted image that fuses the facial attributes of the first object.

[0102] Step 6: Train the image generation model based on the predicted image and the sample reference image.

[0103] Based on this, by training the image generation model, the generation effect of the image generation model can be guaranteed. However, in order to further ensure that the facial attributes of the target image generated by the image generation model are more similar to the facial attributes of the reference image, the sixth step in the above-mentioned image generation model training method can be implemented in the following way: Extract the first image feature vector of the predicted image and the second image feature vector of the sample reference image; The image generation model is trained based on the predicted image, the sample reference image, the first image feature vector, and the second image feature vector.

[0104] In one implementation, a neural network model trained with the ArcFace function is used to extract the first image feature vector of the predicted image and the second image feature vector of the sample reference image. ArcFace is a loss function in deep learning that can enhance the discriminativeness of the model's learned features, making the features of samples of the same class as close as possible in the embedding space, and the features of samples of different classes as far apart as possible. The feature extraction model trained with the ArcFace function can extract the core features of the first object in the predicted image and the sample reference image, so that the extracted first image feature vector and second image feature vector are the core features representing the facial attributes of their corresponding images. By comparing the feature vectors extracted by this method, it is possible to know whether the facial features of the currently generated predicted image are similar to and reproduced with the sample reference image.

[0105] In one implementation, when the first feature processing unit is composed of a preset neural network model, the first feature processing unit of the image generation model can be used directly to extract features from the sample reference image and the predicted image to obtain the first image feature vector of the predicted image and the second image feature vector of the sample reference image. By directly extracting the feature vectors using the first feature processing unit and making subsequent comparisons, the generation effect of the current model can be accurately determined.

[0106] In one implementation, the first image feature vector of the predicted image and the second image feature vector of the sample reference image are first normalized to make the magnitudes of the first image feature vector and the second image feature vector consistent. Then, the similarity between the first image feature vector and the second image feature vector is calculated.

[0107] For example, the similarity calculation method can be to calculate the cosine similarity between the feature vector of the first image and the feature vector of the second image. The cosine similarity calculation method is to divide the dot product of the two vectors by the product of the magnitudes of the two vectors. The range of the cosine similarity calculation result is [-1, 1]. The degree of similarity between the two vectors is judged. The larger the output value, the more similar they are. When the output value is 1, it means that the two vectors are completely identical. Then, the cosine similarity is converted into the corresponding loss value. For the specific formula, please refer to formula (1). L = 1 - similarity (1) That is, subtract the cosine similarity from 1, and finally use the resulting loss value to train the image generation model.

[0108] For example, based on the above calculation of cosine similarity, the loss value can be calculated in the following ways: L=1−cos( θ + mThis means adding an angular interval penalty, where m is a preset angular interval. By introducing a preset angular interval on top of the cosine loss calculation, the first image feature vector and the second image feature vector are forced to be more similar than the ordinary cosine loss.

[0109] Based on this, by comparing the similarity between the feature vectors of the first image and the feature vectors of the second image, and by using the similarity as an aid to train the image generation model, the facial attributes of the target image output by the image generation model can be made to closely resemble the facial attributes of the reference image.

[0110] After training the model, we found that although the model's output correctly captured the facial attributes of the reference image, the output target image was highly blurry. Therefore, this application, based on generative adversarial mechanisms, maintains a judge model during model training to judge and compare the sharpness of the generated target image. The specific implementation process is as follows: Extract the first image sharpness of the predicted image and the second image sharpness of the sample scene image; The image generation model is trained based on the first image sharpness and the second image sharpness.

[0111] For example, the method for extracting sharpness can be a gradient-based method, that is, detecting the intensity changes of the edges and high-frequency regions of the image to be extracted, and calculating the sum of squares of the intensity differences between adjacent pixels. The specific formula is shown in formula (2). (2) Where M and N are the width and height of the image in pixels, I(x,y) is the gray value of the image at position (x,y), and I(x+2,y) is the gray value of the image at position 2 pixels to the right of the current pixel in the same row.

[0112] For example, sharpness can be extracted based on image variance, as shown in formula (3): (3) Where F is the calculated image sharpness evaluation value. The larger the F value, the sharper the image; the smaller the F value, the blurrier the image. M is the width of the image in pixels, N is the height of the image in pixels, I(X,Y) is the pixel value of the image at position (X,Y), and the calculation formula for μ is given in formula (4). (4) This represents the average pixel value of the image.

[0113] In one implementation, a discriminator based on the Generative Adversarial Network standard can be used. In this discriminator, the discriminator is not a pre-trained fixed model, but a dynamic component that is synchronously and alternately optimized with the image generation model during adversarial training.

[0114] After obtaining the first image sharpness of the predicted image and the second image sharpness of the initial image of the sample scene, the realism probability is output by comparing the sharpness difference between the first and second image sharpness. The network weights are adjusted in reverse based on the realism probability, and iterative optimization is performed with the image generation model until convergence. This creates a more realistic output in the image generation model. The discriminator in this application is a dynamic module that is optimized in real time during the training of the image generation model. Its parameter updates are performed alternately with the image generation model, which together constitute the core engine of adversarial training and ensure the sharpness of the target image generated by the image generation model.

[0115] The following is in conjunction with the appendix Figure 2 Taking the application of the image generation method provided in this application to virtual game scenes for face creation as an example, the image generation method will be further explained. Among other things, Figure 2 This application provides a flowchart illustrating a method for character creation in a virtual game arena, according to an embodiment of the present application. Figure 3 The diagram shows the framework of the image generation model corresponding to this embodiment, which specifically includes the following steps: Step 202: Image Acquisition: Obtain the default face image from the game engine (corresponding to...) Figure 3 The initial face image in the image and the real face image input by the user (corresponding to) Figure 3 (Input image in the image).

[0116] Step 204: Feature Extraction: The real face image input by the user is converted into a face vector (corresponding to a facial feature model trained with the ArcFace loss function) using a facial feature model. Figure 3 The ArcFace vector is then processed through a two-layer fully connected layer to obtain the Style vector (corresponding to the ArcFace vector). Figure 3 The `Style` vector in the image is used as the initial image, which is taken from the default face shape in the game engine and input into a preset model for decomposition. The initial image is then transformed from 3 channels to 64 channels using convolutional layers (the convolutional layers process the corresponding...). Figure 3 The image is then downsampled four times (the downsampling operation corresponds to...). Figure 3 In the process of using Feat-down1, Feat-down2, Feat-down3, and Feat-down4, some feature parsing is performed each time, multiplying the number of information channels and reducing the image resolution, ultimately obtaining the feature map of the initial image.

[0117] Step 206: Feature Fusion: The feature maps and style vectors of the initial image are fused using a residual network structure that combines two ada-in (adaptive instance normalization) operations (corresponding to...). Figure 3 The Feature-res1, Feature-res2, Feature-ada1, and Feature-ada2 methods fuse the facial feature style vectors of the real face image input by the user into the feature map of the initial image. The fused image is then subjected to four upsampling operations, the opposite of the downsampling described above (the upsampling operation corresponds to...). Figure 3 In the Feat-up3, Feat-up2, and Feat-up1 segments, the downsampled data at the corresponding resolution is incorporated during the first three upsampling processes (the operation of incorporating downsampled data corresponds to...). Figure 3 The model uses Detail1, Detail2, and Detail3 data from the game engine and facial feature style vectors from real-life images input by the user to learn facial information from real-life images while maintaining the style of the game engine, resulting in better output images.

[0118] Step 208: Target Image Generation: Perform a convolution process on the image features that have undergone 3 upsampling steps (corresponding to...) Figure 3 (Feat-up in the process) to obtain the final output image.

[0119] Step 210: Model Training: During training, a loss function based on pixel perspective, a loss function based on feature vector perspective, and a decisionr model based on generative adversarial mechanism are combined. The pixel perspective loss function compares the difference between the output image and the input real face image pixel by pixel, aiming to align the two images as closely as possible in terms of basic facial shape and features, thus capturing basic facial features effectively. The feature vector perspective loss function converts both the user-input real face image and the output image into facial vectors under the ArcFace model and compares the similarity between these two facial vectors, making the output image closely resemble the input real face image. The decisionr's role is to judge whether the output image is a screenshot from the game engine or an image generated by the model. The decisionr based on generative adversarial mechanism can make the clarity of the output image increasingly closer to the screenshot from the game engine during training.

[0120] Corresponding to the above method embodiments, this application also provides an image generation apparatus embodiment. Figure 4 A schematic diagram of the structure of an image generation apparatus according to an embodiment of this application is shown. Figure 4 As shown, the device includes: The acquisition module 402 is configured to acquire a reference image and an initial image; The extraction module 404 is configured to perform object face analysis on the reference image to obtain the facial attribute features of a first object in the reference image, and to extract features from the initial image based on a second object in the initial image to obtain image features; The fusion module 406 is configured to generate a target image that fuses the facial attributes of the first object based on the facial attribute features and the image features.

[0121] Furthermore, the extraction module 404 is specifically configured to, based on the reference image, invoke the first feature processing unit of the image generation model to obtain the facial attribute features of the first object in the reference image; Accordingly, the extraction module 404 is specifically configured to, based on the initial image, call the second feature processing unit of the image generation model to obtain the image features of the initial image; Accordingly, the fusion module 406 is specifically configured to, based on the facial attribute features and the image features, call the feature fusion unit of the image generation model to obtain fused features, and based on the fused features, call the generation unit of the image generation model to generate a target image that fuses the facial attributes of the first object.

[0122] Furthermore, the first feature processing unit includes a feature extraction layer and a feature conversion layer. Accordingly, the extraction module 404 is specifically configured to, based on the reference image, call the feature extraction layer to obtain the facial feature vector of the first object in the reference image; and based on the facial feature vector, call the feature conversion layer to convert the facial feature vector into facial attribute features of the first object.

[0123] Furthermore, the image generation model also includes a convolutional layer. Accordingly, the extraction module 404 is specifically configured to call the convolutional layer based on the initial image to obtain initial image features with a preset number of channels, and call the second feature processing unit based on the initial image features with the preset number of channels to obtain image features of the initial image.

[0124] Furthermore, the image generation model also includes a residual network unit. Accordingly, the fusion module 406 is specifically configured to call the residual network unit to perform feature fusion based on the facial attribute features and the image features to obtain initial fused features, and to call the feature fusion unit of the image generation model based on the facial attribute features and the initial fused features to obtain fused features.

[0125] Furthermore, the second feature processing unit includes multiple coding layers. Accordingly, the extraction module 404 is specifically configured to, based on the initial image, invoke the multiple coding layers to sequentially obtain multi-layer image features of the initial image.

[0126] Furthermore, the feature fusion unit includes multiple decoding layers. Accordingly, the fusion module 406 is specifically configured to call the multiple decoding layers based on the facial attribute features and the multi-layer image features, and sequentially perform feature fusion decoding to obtain fused features.

[0127] Furthermore, the acquisition module 402 is specifically configured to acquire a sample set, wherein the sample set includes a sample reference image and a sample scene image.

[0128] Accordingly, the extraction module 404 is specifically configured to, based on the sample reference image, call the first feature processing unit to obtain the sample facial attribute features of the first sample object in the sample reference image, and based on the sample scene image, call the second feature processing unit to obtain the sample image features of the sample scene image; Accordingly, the fusion module 406 is specifically configured to, based on the sample facial attribute features and the sample image features, call the feature fusion unit to obtain sample fusion features, based on the sample fusion features, call the generation unit to generate a predicted image that fuses the facial attributes of the first sample object, and based on the predicted image and the sample reference image, train the image generation model.

[0129] Furthermore, the extraction module 404 is specifically configured to extract a first image feature vector of the predicted image and a second image feature vector of the sample reference image; and to train the image generation model based on the predicted image, the sample reference image, the first image feature vector and the second image feature vector.

[0130] Furthermore, the extraction module 404 is specifically configured to extract a first image sharpness of the predicted image and a second image sharpness of the sample scene image; and to train the image generation model based on the first image sharpness and the second image sharpness.

[0131] The above is a schematic scheme of an image generation apparatus according to this embodiment. It should be noted that the technical solution of this image generation apparatus and the technical solution of the image generation method described above belong to the same concept. Details not described in detail in the technical solution of the image generation apparatus can be found in the description of the technical solution of the image generation method described above. Furthermore, the components in the apparatus embodiment should be understood as functional modules necessary to implement each step of the program flow or each step of the method; these functional modules are not actual functional divisions or separations. An apparatus claim defined by such a set of functional modules should be understood as a functional module architecture that primarily implements the solution through the computer program described in the specification, and not as a physical device that primarily implements the solution through hardware.

[0132] Figure 5 A structural block diagram of a computing device 500 according to an embodiment of this application is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0133] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include PSTN (Public Switched Telephone Network), LAN (Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., NIC (Network Interface Controller)), such as an IEEE 802.11 WLAN (Wireless Local Area Network) wireless interface, Wi-MAX (Worldwide Interoperability for Microwave Access) interface, Ethernet interface, USB (Universal Serial Bus) interface, cellular network interface, Bluetooth interface, and NFC (Near Field Communication).

[0134] In one embodiment of this application, the aforementioned components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0135] Computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs (Personal Computers). Computing device 500 can also be a mobile or stationary server.

[0136] The processor 520 is used to execute computer-executable instructions for the image generation method.

[0137] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the image generation method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the image generation method described above.

[0138] An embodiment of this application also provides a computer-readable storage medium storing a computer program / instructions which, when executed by a processor, are used for an image generation method.

[0139] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the image generation method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the image generation method described above.

[0140] An embodiment of this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, are used for an image generation method.

[0141] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the image generation method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the image generation method described above.

[0142] The computer program / instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0143] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0144] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0145] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. An image generation method, characterized in that, include: Obtain the reference image and the initial image; Perform object facial analysis on the reference image to obtain the facial attribute features of the first object in the reference image, and extract features from the initial image based on the second object in the initial image to obtain image features; Based on the facial attribute features and the image features, a target image that fuses the facial attributes of the first object is generated.

2. The method according to claim 1, characterized in that, The step of performing object facial analysis on the reference image to obtain the facial attribute features of the first object in the reference image includes: Based on the reference image, the first feature processing unit of the image generation model is invoked to obtain the facial attribute features of the first object in the reference image; The step of extracting features from the initial image based on the second object in the initial image to obtain image features includes: Based on the initial image, the second feature processing unit of the image generation model is invoked to obtain the image features of the initial image; The step of generating a target image that fuses the facial attributes of the first object based on the facial attribute features and the image features includes: Based on the facial attribute features and the image features, the feature fusion unit of the image generation model is invoked to obtain fused features; Based on the fusion features, the generation unit of the image generation model is invoked to generate a target image that fuses the facial attributes of the first object.

3. The method according to claim 2, characterized in that, The first feature processing unit includes a feature extraction layer and a feature transformation layer; The step of calling the first feature processing unit of the image generation model based on the reference image to obtain the facial attribute features of the first object in the reference image includes: Based on the reference image, the feature extraction layer is invoked to obtain the facial feature vector of the first object in the reference image; Based on the facial feature vector, the feature transformation layer is invoked to convert the facial feature vector into facial attribute features of the first object.

4. The method according to claim 2, characterized in that, The image generation model also includes convolutional layers; The step of calling the second feature processing unit of the image generation model based on the initial image to obtain the image features of the initial image includes: Based on the initial image, the convolutional layer is invoked to obtain initial image features with a preset number of channels; Based on the initial image features with the preset number of channels, the second feature processing unit is invoked to obtain the image features of the initial image.

5. The method according to claim 2, characterized in that, The image generation model also includes a residual network unit; After obtaining the image features of the initial image by calling the second feature processing unit of the image generation model based on the initial image, the method further includes: Based on the facial attribute features and the image features, the residual network unit is invoked to perform feature fusion to obtain initial fused features; The step of calling the feature fusion unit of the image generation model based on the facial attribute features and the image features to obtain fused features includes: Based on the facial attribute features and the initial fusion features, the feature fusion unit of the image generation model is invoked to obtain the fusion features.

6. The method according to claim 2, 4 or 5, characterized in that, The second feature processing unit includes multiple coding layers; The step of calling the second feature processing unit of the image generation model based on the initial image to obtain the image features of the initial image includes: Based on the initial image, the multiple coding layers are invoked to sequentially obtain the multi-layer image features of the initial image.

7. The method according to claim 6, characterized in that, The feature fusion unit includes multiple decoding layers; The step of calling the feature fusion unit of the image generation model based on the facial attribute features and the image features to obtain fused features includes: Based on the facial attribute features and the multi-layer image features, the multiple decoding layers are invoked to sequentially perform feature fusion decoding to obtain fused features.

8. The method according to claim 2, characterized in that, Before the first feature processing unit of the image generation model is invoked based on the reference image to obtain the facial attribute features of the first object in the reference image, the method further includes: Obtain a sample set, wherein the sample set includes sample reference images and sample scene images; Based on the sample reference image, the first feature processing unit is invoked to obtain the sample facial attribute features of the first sample object in the sample reference image; Based on the sample scene image, the second feature processing unit is invoked to obtain the sample image features of the sample scene image; Based on the facial attribute features of the sample and the image features of the sample, the feature fusion unit is invoked to obtain the sample fusion features; Based on the sample fusion features, the generation unit is invoked to generate a predicted image that fuses the facial attributes of the first sample object; The image generation model is trained based on the predicted image and the sample reference image.

9. The method according to claim 8, characterized in that, The step of training the image generation model based on the predicted image and the sample reference image includes: Extract the first image feature vector of the predicted image and the second image feature vector of the sample reference image; The image generation model is trained based on the predicted image, the sample reference image, the first image feature vector, and the second image feature vector.

10. The method according to claim 8 or 9, characterized in that, Also includes: Extract the first image sharpness of the predicted image and the second image sharpness of the sample scene image; The image generation model is trained based on the first image sharpness and the second image sharpness.

11. An image generation apparatus, characterized in that, include: The acquisition module is configured to acquire a reference image and an initial image; The extraction module is configured to perform object face analysis on the reference image to obtain the facial attribute features of a first object in the reference image, and to extract features from the initial image based on a second object in the initial image to obtain image features; The fusion module is configured to generate a target image that fuses the facial attributes of the first object based on the facial attribute features and the image features.

12. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein when the computer programs / instructions are executed by the processor, they implement the method according to any one of claims 1 to 6.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program / instructions that, when executed by a processor, implement the method described in any one of claims 1 to 6.

14. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images