Image processing method, and image processing model training method and device

Abstract

The invention relates to an image processing method, an image processing model training method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: performing feature compression on an object template image to obtain initial object attribute features, and encoding the initial object attribute features step by step to obtain a plurality of object attribute features with different feature sizes; performing feature compression on the identity image to obtain an object identity feature; inputting the object identity feature and each object attribute feature into a feature fusion network to obtain an object fusion feature; the feature fusion network comprises a plurality of feature generation layers with different feature sizes, and the object attribute features are used for being input to the feature generation layers matched with the feature sizes; and carrying out decoding processing with improved resolution on the object fusion feature to obtain a corresponding target object image. By adopting the method, the image processing efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method, an image processing model training method, an apparatus, a computer device, a storage medium, and a computer program product. Background Technology

[0002] With the development of artificial intelligence technology, image processing is being applied in more and more scenarios, such as film and television portrait production and virtual character generation.

[0003] In related technologies, image processing is typically based on user requirements, processing the image to be processed to generate a high-resolution target image that meets the user's needs. However, for high-resolution images, the image processing methods in related technologies suffer from low processing efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide an image processing method, image processing model training method, device, computer equipment, storage medium, and computer program product that can improve image processing efficiency in response to the above-mentioned technical problems.

[0005] This application provides an image processing method. The method includes:

[0006] Retrieve the object template image and identity image;

[0007] The object template image is subjected to feature compression to obtain initial object attribute features, and the initial object attribute features are progressively encoded to obtain multiple object attribute features with different feature sizes; the identity image is subjected to feature compression to obtain object identity features;

[0008] The object identity features and each of the object attribute features are input into a feature fusion network to obtain object fusion features; the feature fusion network includes multiple feature generation layers with different feature sizes, and the object attribute features are used to be input into the feature generation layer with matching feature sizes.

[0009] The object fusion features are then subjected to resolution-enhanced decoding to obtain the corresponding target object image.

[0010] This application also provides an image processing apparatus. The apparatus includes:

[0011] The image acquisition module is used to acquire object template images and identity images;

[0012] The feature extraction module is used to compress the object template image to obtain initial object attribute features, and to encode the initial object attribute features step by step to obtain multiple object attribute features with different feature sizes; and to compress the identity image to obtain object identity features.

[0013] The feature fusion module is used to input the object identity features and each of the object attribute features into the feature fusion network to obtain object fusion features; the feature fusion network includes multiple feature generation layers with different feature sizes, and the object attribute features are used to input into the feature generation layer with matching feature sizes;

[0014] The feature decoding module is used to perform resolution-enhanced decoding processing on the fused features of the object to obtain the corresponding target object image.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps described in the image processing method above.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the image processing method above.

[0017] A computer program product includes a computer program that, when executed by a processor, implements the steps described in the image processing method above.

[0018] The aforementioned image processing method, apparatus, computer equipment, storage medium, and computer program product acquire an object template image and an identity image; perform feature compression on the object template image to obtain initial object attribute features, and progressively encode the initial object attribute features to obtain multiple object attribute features of different feature sizes; perform feature compression on the identity image to obtain object identity features; input the object identity features and each object attribute feature into a feature fusion network to obtain object fusion features; the feature fusion network includes multiple feature generation layers of different feature sizes, with the object attribute features used as input to the feature generation layer with matching feature sizes; and perform resolution-enhancing decoding processing on the object fusion features to obtain the corresponding target object image. Thus, when processing high-resolution object template images and identity images, by performing feature compression on both the object template image and the identity image, the resolution of the identity image and object template image can be reduced while preserving the feature effectiveness of the object template image and the identity image. Furthermore, the compressed object identity features and object attribute features are input into the feature fusion network model. This avoids the feature fusion network model directly processing high-resolution object template images and identity images, thereby reducing the number of parameters learned by the feature fusion network model, shortening the image processing time, and improving image processing efficiency. Next, the object fusion image generated by the feature fusion network model is decoded to improve the resolution of the object fusion image, resulting in a high-quality target object image.

[0019] This application provides a method for training an image processing model. The method includes:

[0020] Obtain a first training image set, which includes training object template images and training identity images;

[0021] The first training image set is input into the initial image processing model for training to obtain the corresponding target image processing model. The target image processing model includes a feature fusion network, which includes multiple feature generation layers with different feature sizes. The feature fusion network is used to obtain training object fusion features based on training object identity features and training object attribute features. The training object identity features are obtained by feature compression of the training identity image. The training object attribute features are obtained by feature compression of the training object template image to obtain initial training object attribute features, and then progressively encoding the initial training object attribute features. The training object attribute features are used to input into the feature generation layer with matching feature sizes.

[0022] The target image processing model is used to process the input object template image and identity image to obtain the corresponding target object image; the target object image is obtained by decoding the object fusion features corresponding to the object template image and the identity image with increased resolution.

[0023] This application also provides an image processing model training apparatus. The apparatus includes:

[0024] An image set acquisition module is used to acquire a first training image set, which includes training object template images and training identity images.

[0025] The model training module is used to input the training object template image and the training identity image into an initial image processing model for training to obtain a corresponding target image processing model. The target image processing model includes a feature fusion network, which includes multiple feature generation layers with different feature sizes. The feature fusion network is used to obtain training object fusion features based on training identity features and training attribute features. The training identity features are obtained by feature compression of the training identity image. The training attribute features are obtained by feature compression of the training object template image to obtain initial object attribute features, and then progressively encoding the initial object attribute features. The training object attribute features are used to input into the feature generation layer with matching feature sizes.

[0026] The target image processing model is used to process the input object template image and identity image to obtain the corresponding target object image; the target object image is obtained by decoding the object fusion features corresponding to the object template image and the identity image with increased resolution.

[0027] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps described in the image processing model training method above.

[0028] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the image processing model training method above.

[0029] A computer program product includes a computer program that, when executed by a processor, implements the steps described in the image processing model training method above.

[0030] The aforementioned image processing model training method, apparatus, computer equipment, storage medium, and computer program product acquire a first training image set, which includes training object template images and training identity images; input the first training image set into an initial image processing model for training to obtain a corresponding target image processing model; the target image processing model includes a feature fusion network, which includes multiple feature generation layers with different feature sizes; the feature fusion network is used to obtain training object fusion features based on training object identity features and training object attribute features, where the training object identity features are obtained by feature compression of the training identity images, and the training object attribute features are obtained by feature compression of the training object template images to obtain initial training object attribute features, and then progressively encoding the initial training object attribute features, which are used as input to the feature generation layer with matching feature sizes; the target image processing model is used to process the input object template images and identity images to obtain a corresponding target object image; the target object image is obtained by decoding the object fusion features corresponding to the object template images and identity images with increased resolution. In this method, training object template images and training identity images are input into an initial image processing model for training. First, feature compression is performed on the training identity images to obtain training object identity features. Then, feature compression is performed on the training object template images to obtain initial training object template features. These initial training object template features are then progressively encoded to obtain multiple training object attribute features of different feature sizes, reducing the resolution of both the training object template images and training identity images. Given the lower resolution of the input images, the feature fusion network requires fewer parameters to learn during the process of obtaining the training object fusion features based on the training object identity features and training object attribute features. This results in faster feature fusion processing, improving the overall training speed of the initial image processing model and yielding the target image processing model. Naturally, the efficiency of processing the object template images and identity images using the target image processing model is also relatively fast. Furthermore, by further decoding the object fusion features corresponding to the object template images and identity images using the target image processing model to improve resolution, the resolution of the output image can be further enhanced, improving the quality of the target object image. Attached Figure Description

[0031] Figure 1 This is an application environment diagram of the image processing method and the image processing model training method in one embodiment;

[0032] Figure 2 This is a flowchart illustrating an image processing method in one embodiment;

[0033] Figure 3 This is a schematic diagram illustrating the result of an image processing method in one embodiment;

[0034] Figure 4 This is a schematic diagram of the network architecture for extracting attribute features in one embodiment;

[0035] Figure 5 This is a schematic diagram of the network architecture for extracting identity features in one embodiment;

[0036] Figure 6 This is a schematic diagram of a network architecture for feature fusion in one embodiment.

[0037] Figure 7 This is a schematic diagram of the network architecture of the encoding / decoding network in one embodiment;

[0038] Figure 8 This is a schematic diagram illustrating the result of a face-swapping processing method for document images in one embodiment.

[0039] Figure 9 This is a flowchart illustrating an image processing model training method in one embodiment;

[0040] Figure 10 This is a schematic diagram of the network architecture of the initial image processing model in one embodiment.

[0041] Figure 11 This is a schematic diagram of the network architecture of a target image processing model in one embodiment;

[0042] Figure 12 This is a schematic diagram of the network architecture for extracting attribute features in one embodiment;

[0043] Figure 13 This is a structural block diagram of an image processing device in one embodiment;

[0044] Figure 14 This is a structural block diagram of an image processing model training device in one embodiment;

[0045] Figure 15 This is an internal structural diagram of a computer device in one embodiment;

[0046] Figure 16 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0049] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0050] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the vision field, such as Swin-transformer, ViT, V-MOE, and MAE, can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0051] The solutions provided in this application involve artificial intelligence processing such as machine learning and computer vision, and can be applied to, but are not limited to, scenarios such as film and television portrait production, game character design, virtual avatars, and information protection. Specific examples are provided below:

[0052] The image processing method and image processing model training method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on the cloud or other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal 102 and server 104 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0053] Both the terminal and the server can be used independently to execute the image processing method and image processing model training method provided in the embodiments of this application.

[0054] For example, the server acquires an object template image and an identity image. It performs feature compression on the object template image to obtain initial object attribute features, and then progressively encodes these initial features to obtain multiple object attribute features of different feature sizes. The server also performs feature compression on the identity image to obtain object identity features. The server inputs the object identity features and each object attribute feature into a feature fusion network to obtain object fusion features. These fusion features are then subjected to resolution-enhanced decoding to obtain the corresponding target object image. The feature fusion network includes multiple feature generation layers with different feature sizes, and the object attribute features are used as input to the feature generation layers with matching feature sizes.

[0055] The server obtains the first training image set, inputs the first training image set into the initial image processing model for training, and obtains the corresponding target image processing model.

[0056] The terminal and server can also work together to execute the image processing method and image processing model training method provided in the embodiments of this application.

[0057] For example, the server obtains a first training image set from the terminal, inputs the first training image set into an initial image processing model for training, and obtains the corresponding target image processing model. The server sends the target image processing model to the terminal, and the terminal processes the object template image and identity image using the target image processing model to obtain the corresponding target object image.

[0058] In one embodiment, such as Figure 2 As shown, an image processing method is provided. Taking the application of this method to a computer device as an example, the computer device can be a terminal or a server. The method can be executed independently by the terminal or server, or it can be implemented through interaction between the terminal and the server. (Reference) Figure 2 The image processing method includes the following steps:

[0059] S202, Obtain the object template image and identity image.

[0060] Image processing refers to adjusting the content of an input image to obtain an adjusted target image. In this embodiment, image processing refers to feature fusion of an object template image and an identity image. Specifically, while keeping the attribute features of the object template image unchanged, the identity features of the object template image are adjusted to match the identity features of the identity image to generate a corresponding target object image. This ensures that the attribute information of the target object image matches the object template image, and the identity information of the target object image matches the identity image.

[0061] An object template image is an image that represents the behavioral characteristics of an object. Object template images include, but are not limited to, images of people and images of animals.

[0062] For example, if the object template image is a person object image, the object template image can include attribute features such as hair features, expression features, angle features, and background features. Among them, hair features are used to describe the hair of the person in the person object image, specifically curly hair, straight hair, short hair, etc.; expression features are used to describe the expression of the person, specifically smiling, laughing, sad, etc.; angle features are used to describe the angle of the person in the person object image, such as side view, front view, back view, etc.; and background features are used to describe the background of the person in the person object image, such as building background, document background, etc.

[0063] An identity image is an image that represents the facial features of an object. Identity images include, but are not limited to, images of people and images of animals.

[0064] Taking the identity image as an example of a person, the identity image can include features such as the eyes, mouth and nose, and eyebrows. Specifically, the eye feature describes the person's eyes, such as black or gray pupils; the mouth and nose feature describes the person's mouth and nose, such as cherry lips or smiling lips; and the eyebrow feature describes the person's eyebrows, such as willow-leaf eyebrows or straight eyebrows.

[0065] It should be noted that the object template image and the identity image are images of the same type. For example, both the object template image and the identity image are images of people, cats, or dogs.

[0066] Specifically, the computer device can obtain the object template image and identity image locally or from other devices, and obtain the target object image by processing the object template image and identity image.

[0067] In one embodiment, the object features represented in the object template image and the identity image may overlap. For example, the object template image may include both object behavioral features and facial features, and the identity image may also include both object behavioral features and facial features.

[0068] S204, perform feature compression on the object template image to obtain initial object attribute features, and encode the initial object attribute features step by step to obtain multiple object attribute features with different feature sizes; perform feature compression on the identity image to obtain object identity features.

[0069] Feature compression is used to compress the features of an image, resulting in an image with reduced resolution to some extent while retaining important features.

[0070] For identity images, feature compression is used to extract various features from the image. Then, object identity features are selected from the extracted features and fused to obtain a more compact set of object identity features. The resolution of the object identity features is lower than the resolution of the identity image. It should be understood that one identity image corresponds to one set of object identity features.

[0071] For the object template image, feature compression is used to extract various features from the object template image and filter attribute features from the extracted feature information as initial object attribute features. The resolution of the initial object attribute features is lower than the resolution of the object template image.

[0072] In real-world scenarios, considering the diverse types of attribute features of object template images, after obtaining the initial object attribute features, the initial object attribute features are further encoded step by step to achieve feature fusion and obtain more compact object attribute features.

[0073] Specifically, the initial object attribute features are input into alternating convolutional and downsampling layers, progressively encoding them into first object attribute feature maps of different feature sizes. These first object attribute feature maps are then passed through alternating convolutional and upsampling layers, progressively decoding them into second object attribute features of different feature sizes. The object attribute features can be the first object attribute feature map obtained only through alternating convolutional and downsampling layers, or the first object attribute features obtained through alternating convolutional, downsampling, convolutional, and upsampling layers. This application does not impose any limitations. It should be understood that one object template image corresponds to one set of initial object attribute features, as well as multiple sets of object attribute features.

[0074] In one scenario, an object template image and an identity image are input into two identical feature compression models. The identity image is compressed using the feature compression model to obtain object identity features, and the object template image is compressed using the same model to obtain initial object attribute features. Then, these initial object attribute features are encoded layer by layer using alternating convolutional and upsampling layers to obtain multiple object attribute features with different feature sizes.

[0075] In another scenario, the object template image and the identity image are input into different feature compression models. The different feature compression models are used to compress the features of the different images to obtain the corresponding features.

[0076] Specifically, the object template image is input into the first feature compression model to obtain object attribute features of multiple different feature sizes; the identity image is input into the second feature compression model to obtain object identity features.

[0077] S206, The object identity features and the object attribute features are input into the feature fusion network to obtain the object fusion features; the feature fusion network includes multiple feature generation layers with different feature sizes, and the object attribute features are used as inputs to the feature generation layer with matching feature sizes.

[0078] Among them, the feature fusion network is a deep learning model pre-trained by a computer device based on a training image set, which includes multiple feature generation layers with different feature sizes.

[0079] In this embodiment of the application, the feature fusion network fuses the input object identity features and object attribute features to obtain object fusion features, so that the attribute features of the object fusion features match the object attribute features of the object template image, and the identity features of the object fusion features match the object identity features of the identity image.

[0080] In an optional embodiment, the feature fusion network includes multiple cascaded feature generation layers, with the feature sizes of adjacent feature layers increasing sequentially.

[0081] The computer device determines the object attribute features input to each feature generation layer based on the feature size of each feature generation layer. Then, each object attribute feature is input to the feature generation layer with matching feature size, and the object identity features are input to each feature generation layer. The input object attribute features and object identity features are processed sequentially by each feature generation layer in the feature fusion network to obtain the object fusion features.

[0082] It should be noted that in the feature fusion network, the input of the next feature output layer includes not only the object identity features and the object attribute features that match the feature size, but also the output of the previous feature generation layer.

[0083] Specifically, based on the acquisition of multiple object attribute features, the first object attribute feature of the first feature generation layer is determined according to the feature size of the first feature generation layer in the feature fusion network. The object identity feature and the first object attribute feature are then fused through the first feature generation layer to obtain the first intermediate fused feature. Next, the second object attribute feature matching the feature size of the second feature generation layer is determined according to the feature size of the second feature generation layer in the feature fusion network. The object identity feature, the second object attribute feature, and the first intermediate fused feature are then fused through the second feature generation layer to obtain the second intermediate fused feature. This process is repeated until the object fusion feature output by the last feature generation layer in the feature fusion network is obtained.

[0084] S208, perform resolution-enhanced decoding on the object fusion features to obtain the corresponding target object image.

[0085] As the preceding steps show, the resolution of object identity features is lower than that of the identity image, and the resolution of object attribute features is lower than that of the object template image. Therefore, the resolution of the object fusion feature synthesized from the low-resolution object identity features and various object attribute features is naturally also lower than that of the identity image and the object template image. Based on this, in order to restore the resolution of the object fusion feature as much as possible, it is necessary to decode the object fusion feature to improve image quality.

[0086] In this embodiment, when the resolution of the target object image is higher than that of the object attribute features and the object identity features, the resolution of the target object image and the resolution of the object template image are not limited.

[0087] In one embodiment, a computer device acquires a decoding network, inputs object fusion features into the decoding network, and gradually decodes the object fusion features through the decoding network to obtain an image of the target object.

[0088] Specifically, the resolution of the object template image is used as the target resolution. The object fusion features are gradually decoded through a decoding network until the resolution of the object fusion image is increased to the target resolution, thus obtaining the target object image.

[0089] It should be noted that since the target object image is obtained based on object fusion feature decoding, if the attribute features of the object fusion feature match the object attribute features of the object template image, and the identity features of the object fusion feature match the object identity features of the identity image, then the attribute features of the target object image also match the object template image, and the identity features of the target object image match the identity image.

[0090] In practical applications, the image processing operations of processing the object template image and identity image to obtain the target object image in this embodiment can be integrated into a pre-trained target image processing model. The target image processing model is a deep learning model, specifically trained by a computer device on an initial image processing model based on a first training image set. The detailed training process will be explained in subsequent embodiments regarding image processing model training, and will not be repeated here.

[0091] Furthermore, the application process of the image processing model is explained. The input images of the image processing model are the object template image and the object identity image, and the output is the target object image. Please refer to [link / reference needed]. Figure 3 , Figure 3 The diagram illustrates the input and output images of the target image processing model. As can be seen from the diagram, the attribute features of the target object image output by the target image processing model match the object template image, and the identity features of the target object image match the identity image.

[0092] In the above image processing method, an object template image and an identity image are acquired; the object template image is feature-compressed to obtain initial object attribute features, and these initial object attribute features are progressively encoded to obtain multiple object attribute features of different feature sizes; the identity image is feature-compressed to obtain object identity features; the object identity features and each object attribute feature are input into a feature fusion network to obtain object fusion features; the feature fusion network includes multiple feature generation layers of different feature sizes, with object attribute features used as input to feature generation layers matching the feature size; the object fusion features are then decoded to improve resolution, resulting in the corresponding target object image. Thus, when processing high-resolution object template images and identity images, feature compression of both the object template image and the identity image reduces their resolution while preserving the effectiveness of their features. Furthermore, inputting the feature-compressed object identity features and object attribute features into the feature fusion network model avoids the model directly processing the high-resolution object template image and identity image, thereby reducing the number of parameters the feature fusion network model learns, shortening its image processing time, and improving image processing efficiency. Next, the object fusion image generated by the feature fusion network model is decoded to improve the resolution of the object fusion image and obtain a high-quality target object image.

[0093] In one embodiment, feature compression is performed on the object template image to obtain initial object attribute features, including:

[0094] The object template image is compressed using the first encoding network to extract attribute features, resulting in an object attribute feature distribution. Object attribute features that conform to a normal distribution are selected from the object attribute feature distribution as initial object attribute features.

[0095] The first encoding network is a deep learning network pre-trained on the second training image set, which includes a series of convolutional and pooling layers to extract initial attribute features of the object template image.

[0096] When training the first encoding network, the first encoding network can be trained as an independent network, or the first initial encoding network can be combined with the initial decoding network to generate an initial encoding-decoding network. After the initial encoding-decoding network is trained, the first encoding network can be separated from the trained target encoding-decoding network.

[0097] For example, a computer device concatenates a first initial encoding network and an initial decoding network to obtain an initial encoding-decoding network. Then, based on a second training image set constructed from training samples and training labels, the initial encoding-decoding network is trained as a whole to obtain a trained encoding-decoding network, and the first encoding network is extracted from the encoding-decoding network.

[0098] refer to Figure 4 , Figure 4 This is a schematic diagram of the architecture of the first coding network. Figure 4 In this process, the first encoding network filters out basic attribute features related to the object from multiple features in the object template image, compresses the basic attribute features to obtain attribute features, and then uses a normal distribution feature extraction method to sample object attribute features that conform to the normal distribution from the object attribute feature distribution corresponding to the attribute features, i.e., the initial object attribute features.

[0099] In the above embodiments, feature compression is performed on the object template image to extract attribute features, resulting in an object attribute feature distribution. This initially eliminates redundant features in the object template image while preserving the effectiveness of the object attribute features. Further, object attribute features conforming to a normal distribution are selected from the object attribute feature distribution as initial object attribute features. This further refines the elimination of redundant features, improves the effectiveness of the object attribute features, and thus enhances computational efficiency. Moreover, the normally distributed initial object features extracted using the normalization extraction method possess additivity, multiplicativeness, and linear transformation properties, facilitating subsequent deep computation on the initial object attribute features and improving computational speed.

[0100] In one embodiment, feature compression is performed on the identity image to obtain the object's identity features, including:

[0101] The identity image is transformed by a second coding network to obtain the initial object identity features; the initial object identity features are then nonlinearly mapped to obtain the object identity features.

[0102] The second encoding network is a deep learning network pre-trained on the third training image set. It is used to perform feature transformation on the identity image to obtain more compact initial object identity features in the identity image.

[0103] Specifically, the third training image set includes training images and training feature labels. The computer device inputs the training images into the second initial encoding network to obtain training identity features. Based on the feature loss of the training identity features and training feature labels, the network parameters in the second initial encoding network are optimized until the second initial encoding network converges to obtain the second encoding network.

[0104] In practical applications, please refer to Figure 5The identity image is input into a second encoding network, which performs feature dimensionality reduction on the identity image to obtain initial object identity features. It should be noted that the resolution of the initial object identity features is much lower than the resolution of the identity image.

[0105] Considering the complex correlations between features in the identity image, in order to further explore the correlations and complexity between the initial object identity features in the identity image, a nonlinear mapping is performed on the initial object identity features to obtain the object identity features.

[0106] Please continue to refer to this. Figure 5 The initial object identity features are input into a mapping network that includes multiple fully connected layers, and nonlinear mapping of the initial object identity features is performed to obtain the object identity features.

[0107] In the above embodiments, the identity image is subjected to feature dimensionality reduction through a second encoding network to eliminate redundant attribute features while retaining the identity features, thus ensuring that the initial identity features are both effective and compact. Next, a nonlinear mapping is applied to the initial object identity features to enhance their nonlinear expressive power.

[0108] In one embodiment, object identity features and various object attribute features are input into a feature fusion network to obtain object fusion features, including:

[0109] Object attribute features of different feature sizes are input into feature generation layers with matching feature sizes; object identity features are input into each feature generation layer; for any feature generation layer, the object identity features are transformed by the feature generation layer to obtain transformed identity features that match the feature size of the feature generation layer; the transformed identity features, object attribute features, and the output features of the previous feature generation layer are superimposed to obtain the initial object fusion features; the initial object fusion features are convolved to obtain the output features corresponding to the feature generation layer; the output features of the last feature generation layer in the feature fusion network are determined as the object fusion features.

[0110] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the network architecture of a feature fusion network in one embodiment. Figure 6 The feature fusion network shown includes four feature generation layers. Figure 6The four feature generation layers shown have feature sizes of 8×8, 16×16, 32×32, and 64×64 from left to right. Specifically, the inputs to the 8×8 feature generation layer are object identity features and object attribute features (8×8 each); the inputs to the 16×16 feature generation layer are object identity features and object attribute features (16×16 each); the inputs to the 32×32 feature generation layer are object identity features and object attribute features (32×32 each); and the inputs to the 64×64 feature generation layer are object identity features and object attribute features (64×64 each).

[0111] In feature fusion networks, each feature generation layer operates identically except for its input features. The core operations are upsampling, feature size variation, feature stacking, and convolution. Furthermore, the input and output feature sizes of each feature generation layer remain unchanged after processing by that layer.

[0112] Specifically, taking a feature generation layer with a feature size of 16×16 as an example, the process of the feature generation layer processing the input object identity features and object attribute features (16×16) with matching feature size is explained as follows: The output (8×8) of the previous feature generation layer is upsampled and used as the overall output feature of the previous layer (16×16). The feature size of the object identity features is transformed into the transformed identity features (16×16) with the same size as the feature generation layer through a style transfer model. The output features, transformed identity features and object attribute features of the previous layer are stacked element by element to obtain the initial object fusion features. Finally, the initial object fusion features are convolved and output to the next feature generation layer (16×16).

[0113] The feature generation layers of the feature fusion network process the above-mentioned processing on the output of the previous feature generation layer and the input object attribute features and object identity features in sequence until the last feature generation layer is completed. The output feature of the last feature generation layer is then determined as the object fusion feature.

[0114] In the above embodiments, each feature generation layer in the feature fusion network converts the feature size of the object identity feature to match the feature size of the feature generation layer, and then superimposes it with the object attribute feature to fully fuse the object attribute feature and the object identity feature. Furthermore, by using multiple feature generation layers with different feature sizes, object attribute features of different feature sizes are processed to fuse object attribute features and object identity features from multiple dimensions, thereby improving the comprehensiveness and richness of the fused object features.

[0115] In one embodiment, the object fusion features are decoded with increased resolution to obtain the corresponding target object image, including:

[0116] The object fusion features are upsampled by a decoding network to obtain the target object image; the resolution of the target object image matches the resolution of the object template image.

[0117] The decoding network can be concatenated with the first encoding network and pre-trained based on a second training image set. Since the decoding network and the first encoding network are trained as a whole, the training process of the decoding network naturally overlaps with that of the first encoding network.

[0118] For example, a computer device concatenates the first initial encoding network and the initial decoding network to obtain the initial encoding / decoding network. Please refer to [reference needed]. Figure 7 , Figure 7 This is a schematic diagram of the architecture of an encoding / decoding network in one embodiment. The encoding network in this encoding / decoding network can be... Figure 4 The first encoding network shown is the decoding network in this embodiment. Under this architecture, the computer device trains the initial encoding / decoding network as a whole based on the second training image set constructed from training samples and training labels, obtaining the trained encoding / decoding network, and extracting the decoding network from the encoding / decoding network.

[0119] Furthermore, the decoding network includes multiple convolutional layers and downsampling layers. Through a series of convolutional layers and downsampling layers, the object fusion features are decoded layer by layer, thereby improving the resolution of the object fusion features in turn. The output of the last downsampling layer in the decoding network is used as the target object image.

[0120] It should be understood that the purpose of the first encoding network is to reduce the resolution of the object template image, while the purpose of the decoding network is to improve the resolution of the object fusion features. In other words, the decoding network and the first encoding network are symmetrical to each other. Therefore, the resolution of the target object image obtained by the decoding network is also matched with the resolution of the object template image before the first encoding network performs encoding processing. For example, the resolution of the target object image is the same as the resolution of the object template image, which is 1024×1024.

[0121] In the above embodiments, for low-resolution object fusion features, the object fusion features are upsampled through a decoding network to gradually increase the resolution of the object fusion features until it matches the resolution of the object template image, so that the resolution of the images before and after the entire image processing is as close as possible, and a high-quality target object image is obtained.

[0122] In one embodiment, the image processing method further includes, before feature compression of the object template image:

[0123] Object recognition is performed on the object template image to obtain the first region of interest (ROI) of the object template image, and image registration is performed on the first ROI; object recognition is performed on the identity image to obtain the second ROI of the identity image, and image registration is performed on the second ROI; the resolution of the image-registered object template image and the image-registered identity image is normalized.

[0124] Object recognition refers to cropping or defining the region of interest (ROI) containing the target object in the input image, such as face recognition. Image registration refers to aligning and adjusting the target object within the ROI.

[0125] For example, if the object being identified is a face and the region of interest is the face region, image registration can be done by aligning the eyebrows horizontally and the mouth and nose vertically.

[0126] For example, object recognition is face recognition. Computer devices can use an object recognition model to recognize the input image, and then use an image registration model to register the region of interest identified by the face.

[0127] Specifically, the face recognition model is used to perform face recognition on the object template image to obtain the face template region, which is the first region of interest. Then, the local areas in the face template region are fine-tuned by the image registration model, such as aligning the eyes horizontally and the mouth and nose vertically, to obtain the object template image after image registration.

[0128] Specifically, the identity image is subjected to face recognition by a face recognition model to obtain the face region to be fused, which is the second region of interest. Then, the local region in the face region to be fused is fine-tuned by an image registration model to obtain the identity image after image registration.

[0129] It should be noted that the image registration of the first region of interest in the object template image and the image registration of the second region of interest in the identity image are performed independently by the computer equipment.

[0130] Furthermore, to facilitate the subsequent extraction and fusion calculation of features in the identity image and the object template image, the resolution of the object template image after image registration and the resolution of the identity image after image registration can be further adjusted to a fixed resolution threshold.

[0131] For example, keep the resolution of the object template image after image registration unchanged, and adjust the resolution of the identity image after image registration until the two have the same resolution.

[0132] For example, a target resolution is set, and the resolution of the object template image and the identity image after image registration are adjusted respectively, so that the adjusted resolutions are the same and both are the target resolution.

[0133] In the above embodiments, after obtaining the object template image and the identity image, before performing feature compression on the object template image and the identity image, region of interest identification and image registration are performed on the object template image and the identity image respectively, and the resolution of the image-registered object template image and the image-registered identity image are unified, so as to facilitate the rapid extraction and fusion of the features of the two in the future, thereby improving the image processing speed.

[0134] In one embodiment, the object template image is any image from the object template video; the image processing method further includes:

[0135] Obtain the object attribute features of each object template image; fuse the object identity features with the object attribute features of each object template video to obtain multiple object fusion features; perform resolution-enhanced decoding on each object fusion feature to obtain the corresponding target object image; and obtain the corresponding target video based on each target object image.

[0136] In this context, an object template video is typically a continuous video of a single object, which can be a movie or a virtual video, etc. If the object's identity in the object template video does not meet the requirements, the object's identity features can be obtained from the user-provided identity image, and then replaced with the object's identity features from the identity image.

[0137] In the processing scenario of object template videos, each frame of the object template video serves as an object template image, and the identity image corresponding to each object template image is the same. That is, one object template video corresponds to one identity image to maintain the consistency of the object identity in the object template video.

[0138] After acquiring the object template video, the object template images are extracted from the video, and feature compression is performed on each object template image to obtain multiple sets of object attribute features, with one object template image corresponding to one set of object attribute features. Then, the same object identity feature is fused with different object attribute features to obtain multiple sets of object fused features. Next, each set of object fused features is decoded, and the resolution of each object fused feature is adjusted to match the object template image in each frame of the object template video, resulting in multiple target object images. Finally, all target object images are combined to obtain the target video.

[0139] It should be understood that the identity features of each target object image in the target video are matched with the same identity image, and the facial expression features, angle features, and background features of each target object image in the target video are matched with different identity template images.

[0140] In the above embodiments, feature fusion is performed separately on the same identity image and each object template image in the object template video. The resulting multiple object fused features share the identity features of a single identity image, maintaining the consistency of object identities in the object template video and improving the quality of the target video. Furthermore, the above embodiments process each object identity feature in the object template video in the same way, supporting parallel processing of different object template images, thereby improving the processing efficiency of the target video.

[0141] In one embodiment, the object template image is an ID card template image; the target object image is a target ID card image, the identity features of the target ID card image are matched with the identity image, and the facial expression features, angle features, and background features of the target ID card image are matched with the ID card template image.

[0142] When the object template image is an ID card template image, the identity image can be a user's face image provided by the user based on actual needs, and the feature fusion model is a face fusion model.

[0143] In the context of face-swapping for ID photos, the target ID image is obtained by processing the ID template image and the user's facial image. Please refer to [reference needed]. Figure 8 , Figure 8 This is a schematic diagram showing the image processing results before and after face-swapping in an ID photo scenario.

[0144] In practical applications, computer equipment compresses the features of the document template image, extracts the facial expression features, angle features, and background features from the document template image, compresses the features of the user's face image, extracts the user identity features from the user's face image, and then uses a feature fusion model to fuse the user identity features of the user's face image with the facial expression features, angle features, and background features from the document template image, and improves the resolution of the fused document features to obtain the target document image.

[0145] Specifically, by compressing the features of the document template image, initial document object attribute features are obtained. These initial document object attribute features are then progressively encoded to obtain multiple document object attribute features of different feature sizes. Each set of document object attribute features includes facial expression features, angle features, and background features from the document template image. The user's face image is then compressed to obtain user identity features. The user identity features and multiple document object attribute features are input into the face fusion model to obtain document fusion features. The document fusion features are then decoded to improve their resolution, resulting in the target document image.

[0146] In this embodiment, facial expression features, angle features, and background features in the ID card template image are determined as object attribute features of the target ID card image, and object identity features in the identity image are determined as identity features of the target ID card image, thus obtaining a target object image that meets the fusion requirements. This processing logic has strong portability, supports the fusion of different types of ID card template images with any identity image, and can be widely applied in various practical ID photo face-swapping scenarios.

[0147] In one embodiment, such as Figure 9 As shown, an image processing model training method is provided. Taking the application of this method to a computer device as an example, the computer device can be a terminal or a server. This method can be executed independently by the terminal or server, or it can be implemented through interaction between the terminal and the server. (Reference) Figure 9 The image processing model training method includes the following steps:

[0148] S902, Obtain the first training image set, which includes training object template images and training identity images.

[0149] In the first training image set, the training object template images and training identity images are images of the same category, such as all being human images, all being cat images, or all being dog images, etc.

[0150] It should be understood that both the training object template image and the training identity image are input images of the initial image processing model, and the true output of the initial image processing model is the training object template image.

[0151] Specifically, the computer device can obtain a first training image set and an initial image processing model locally or from other devices, and train the initial image processing model using the first training image set to obtain the image processing model.

[0152] S904, the first training image set is input into the initial image processing model for training to obtain the corresponding target image processing model; the target image processing model includes a feature fusion network, which includes multiple feature generation layers with different feature sizes; the feature fusion network is used to obtain training object fusion features based on the training object identity features and training object attribute features. The training object identity features are obtained by feature compression of the training identity image, and the training object attribute features are obtained by feature compression of the training object template image to obtain the initial training object attribute features, and the initial training object attribute features are encoded step by step. The training object attribute features are used to input into the feature generation layer with matching feature sizes; the target image processing model is used to process the input object template image and identity image to obtain the corresponding target object image; the target object image is obtained by decoding the object fusion features corresponding to the object template image and identity image with increased resolution.

[0153] The initial image processing model takes a training object template image and a training identity image as input, and outputs a training object image. The feature fusion network in the initial image processing model takes training object identity features and training object attribute features as input, and outputs fused training features. In other words, the initial image processing model includes not only feature fusion via the feature fusion network, but also preprocessing operations before the feature fusion network, such as feature compression of the training identity image to obtain training object identity features, feature compression of the training object template image, and progressive encoding to obtain training object attribute features, as well as post-processing operations after the feature fusion network output, such as resolution enhancement of the fused training object features.

[0154] The training object template image and the training identity image from the first training image set are both input into the initial image processing model for training, and the training object image output by the initial processing model is obtained. Based on the attribute feature loss between the training object image and the training object template image, and the identity feature loss between the training object image and the training object template image, the parameters of the initial image processing model are optimized until the initial image processing model converges, and the target image processing model is obtained.

[0155] In this way, when training the initial image processing model, both the training object template image and the training identity image from the first training image set are input into the initial image processing model for training, and the training object image output by the initial processing model is obtained. Based on the attribute feature loss between the training object image and the training object template image, and the identity feature loss between the training object image and the training object template image, the parameters of the initial image processing model are optimized until the initial image processing model converges, and the target image processing model is obtained.

[0156] The target image processing model includes a feature fusion network, which comprises multiple feature generation layers with different feature sizes. The feature size of the feature generation layer matches the feature size of the input training object attribute features.

[0157] During the training of the initial image processing model, the input to the feature fusion network consists of training object identity features and training object attribute features, and the output is the training object fusion feature. Since the training object identity features are obtained by feature compression of the training identity image, and the training object attribute features are obtained by feature compression of the training object template image, and then progressively encoded, the input to the feature fusion network is essentially feature-compressed, i.e., low-resolution training object identity and attribute features. This avoids the feature fusion network directly learning from high-resolution images, reduces the number of parameters the network learns, and thus prevents overfitting, improving the robustness of the feature fusion network. Furthermore, after obtaining the low-resolution training object fusion feature output by the feature fusion network, resolution upscaling can be performed on this feature to obtain a training object image with a resolution matching the training object template image, improving the image generation quality of the initial image processing model.

[0158] In the above image processing model training method, a first training image set is obtained and input into an initial image processing model for training to obtain a corresponding target image processing model. The target image processing model includes a feature fusion network, which comprises multiple feature generation layers with different feature sizes. The feature fusion network is used to obtain training object fusion features based on training object identity features and training object attribute features. The training object identity features are obtained by feature compression of the training identity image, and the training object attribute features are obtained by feature compression of the training object template image to obtain initial training object attribute features, which are then progressively encoded. These initial training object attribute features are input into the feature generation layer with matching feature sizes. The target image processing model processes the input object template image and identity image to obtain a corresponding target object image. The target object image is obtained by decoding the object fusion features corresponding to the object template image and identity image with increased resolution. In this method, training object template images and training identity images are input into an initial image processing model for training. First, feature compression is performed on the training identity images to obtain training object identity features. Then, feature compression is performed on the training object template images to obtain initial training object template features. These initial training object template features are then progressively encoded to obtain multiple training object attribute features of different feature sizes, reducing the resolution of both the training object template images and training identity images. Given the lower resolution of the input images, the feature fusion network requires fewer parameters to learn during the process of obtaining the training object fusion features based on the training object identity features and training object attribute features. This results in faster feature fusion processing, improving the overall training speed of the initial image processing model and yielding the target image processing model. Naturally, the efficiency of processing the object template images and identity images using the target image processing model is also relatively fast. Furthermore, by further decoding the object fusion features corresponding to the object template images and identity images using the target image processing model to improve resolution, the resolution of the output image can be further enhanced, improving the quality of the target object image.

[0159] In one embodiment, the target image processing model further includes a first encoding network and a decoding network; before acquiring the first training image set, the method further includes:

[0160] A second training image set is obtained, which includes training samples and training labels. The training samples are input into a first initial encoding network to obtain sample object attribute features. The sample object attribute features are input into an initial decoding network to obtain sample training results. The similarity loss is obtained by calculating the difference between the feature distribution of the sample object attribute features and the standard normal distribution. The reconstruction loss is obtained by calculating the feature norm of the sample training results and the training labels. The reconstruction loss and the similarity loss are superimposed to obtain the encoding and decoding loss. The first initial encoding network and the initial decoding network are trained simultaneously based on the encoding and decoding loss to obtain the first encoding network and the decoding network.

[0161] Training samples are input into the first initial encoding network to obtain sample object attribute features. These sample object attribute features are then input into the initial decoding network to obtain sample training results. The similarity loss is obtained by calculating the difference between the feature distribution of the sample object attribute features and the standard normal distribution. The reconstruction loss is obtained by calculating the feature norm of the sample training results and the training labels. The reconstruction loss and the similarity loss are then superimposed to obtain the encoding-decoding loss. The first initial encoding network and the initial decoding network are trained simultaneously based on the encoding-decoding loss until the initial encoding-decoding network converges, resulting in the first encoding network and the decoding network.

[0162] Please continue to refer to the above. Figure 7 The diagram shows a combined network architecture of the first encoding network and the decoding network. Taking a training sample and training label resolution of 1024×1024 as an example, the training process of the first initial encoding network and the initial decoding network is explained as follows: The training samples are input into the first initial encoding network, which outputs a 64×64 object attribute feature distribution. Then, each attribute feature value in the object attribute feature distribution of the first initial encoding network is sampled according to a standard normal distribution to obtain the initial object attribute features. The initial decoding network performs convolution and upsampling operations on the initial object attribute features to output a 1024×1024 sample training result.

[0163] During the training process described above, the feature norms of the training results and training labels are calculated to obtain the reconstruction loss. The formula for calculating the reconstruction loss is as follows:

[0164]

[0165] Where reconstruction loss represents the reconstruction loss, and x represents the sample training result. Indicates training labels, This represents the L2 norm loss operation, which involves first taking the difference between vector elements, then summing the squares, and finally taking the square root.

[0166] The similarity loss is obtained by calculating the difference between the feature distribution of the sample object's attribute features and the standard normal distribution. The formula for calculating the similarity loss is as follows:

[0167]

[0168] Here, similarity loss represents the similarity loss. This represents the feature distribution of the attribute features of the sample object, where N(0,1) represents the standard normal distribution, and D... KL This indicates that an operation based on the probability distribution difference will be performed.

[0169] The reconstruction loss and similarity loss are combined to obtain the encoding / decoding loss. The formula for calculating the encoding / decoding loss (VAE loss) is as follows:

[0170] VAE loss = reconstruction loss + similarity loss

[0171] The first initial encoding network and the initial decoding network are trained simultaneously based on the encoding and decoding loss until the initial encoding and decoding network converges, thus obtaining the first encoding network and the decoding network.

[0172] In the above embodiments, the similarity loss is obtained by calculating the difference between the feature distribution of the sample object attribute features obtained by the first initial encoding network and the standard normal distribution, so as to characterize the training loss of the first initial encoding network; the reconstruction loss is obtained by calculating the feature norm of the training results of the computational samples and the training labels obtained by the initial decoding network, so as to characterize the overall initial decoding capability and the overall loss of the first initial encoding network; the encoding and decoding loss obtained by superimposing the reconstruction loss and the similarity loss can comprehensively and objectively reflect the overall training effect of the first initial encoding network and the initial decoding network, and the effect of training the first initial encoding network and the initial decoding network simultaneously based on the encoding and decoding loss is also better.

[0173] In one embodiment, the initial image processing model includes a first encoding network, a second encoding network, a decoding network, an initial feature fusion network, and an initial discriminant network; the first training image set is input into the initial image processing model for training to obtain the corresponding target image processing model, including:

[0174] The training object template image is input into the first encoding network to obtain the training object attribute features. The training identity image is input into the second encoding network to obtain the training object identity features. The training object attribute features and training object identity features are input into the initial feature fusion network to obtain the training object fusion features. The training object fusion features are input into the decoding network to obtain the corresponding target training object image. The pixel reconstruction loss between the training object template image and the target training object image is calculated. The feature error norm between the training object template image and the target training object image is calculated to obtain the perceptual loss. The feature similarity between the training identity image and the target training object image is calculated to obtain the identity feature loss. The adversarial loss is obtained based on the authenticity judgment result of the initial discriminator network on the target training object image. The global loss of the initial image processing model is obtained based on the pixel reconstruction loss, perceptual loss, identity feature loss, and adversarial loss. The parameters of the initial feature fusion network and the initial discriminator network in the initial image processing model are trained based on the global loss to obtain the target image processing model.

[0175] refer to Figure 10 , Figure 10 This is a schematic diagram of the initial image processing model architecture. Figure 10 The initial image processing model shown includes a first encoding network, a second encoding network, a decoding network, an initial feature fusion network, and an initial discriminator network. The first encoding network and the decoding network can be trained on a second training image set, while the second encoding network is trained on a third training image set. In other words, during the training of the initial image processing model, the parameters of the first encoding network, the second encoding network, and the decoding network can be kept unchanged, while the parameters of the initial feature fusion network and the initial discriminator network can be optimized to obtain the feature fusion network and the discriminator network.

[0176] Specifically, the training object template image is input into the first encoding network to obtain the training object attribute features, and the training identity image is input into the second encoding network to obtain the training object identity features. Next, the training object attribute features and training object identity features are input into the initial feature fusion network to obtain the training object fused features, which are then input into the decoding network to obtain the corresponding target training object image. The target training object image is then input into the initial discriminator network to obtain the authenticity judgment result of the target training object image, which is represented by "0" or "1". When training the initial image processing model based on the training object template image, the initial image processing model outputs the target training object image and the authenticity judgment result.

[0177] Based on this, the global loss of the initial image processing model can be determined by calculating the loss between the training object template image and the target training object image, the loss between the training identity image and the target training object image, and the adversarial loss of the target training object image. Based on this, the initial image processing model can be trained until the model converges, and a trained model can be obtained.

[0178] The loss between the training template image and the target training image includes pixel reconstruction loss and perceptual loss. In this embodiment, the purpose of using pixel reconstruction loss and perceptual loss for training is to make the training template image and the target training image as close as possible.

[0179] The pixel reconstruction loss is used to characterize the pixel-level differences between the training template image and the target training image, and the calculation formula is as follows:

[0180] Reconstruction_loss = abs (fake–GT)

[0181] Wherein, Reconstruction_loss represents pixel reconstruction loss, fake represents the target training object image, GT represents the training object template image, and abs() represents performing the operation of taking the absolute value of the pixel difference.

[0182] Perceptual loss is used to characterize the difference between the training template image and the target training image at the feature level, and can be represented by LPIPS_loss. In this embodiment, since perceptual loss can characterize feature differences, there are no restrictions on the calculation formula of perceptual loss. For example, it can use Euclidean distance between features, Manhattan distance between features, Mahalanobis distance between features, etc.

[0183] The loss between the training identity image and the target training object image is the identity feature loss, obtained by calculating the feature similarity between the two images, and can be represented by ID_loss. In this embodiment, the purpose of using identity feature loss for training is to make the training identity image and the target training object image as similar as possible.

[0184] The adversarial loss is calculated based on the judgment result obtained by the initial discriminant network on the authenticity of the target training object image. In this embodiment, the purpose of using the adversarial loss for training is to optimize the parameters of the initial discriminant network and improve the generative adversarial learning effect of the feature fusion network and the discriminant network.

[0185] The formula for calculating adversarial losses is as follows:

[0186] G_loss = log ( 1–D(fake))

[0187] Where G_loss represents the adversarial loss of the feature fusion network, and D(fake) represents the result of the discriminant network judging the authenticity of the target training object image, usually represented by "0" or "1".

[0188] After obtaining the pixel reconstruction loss, perceptual loss, identity feature loss, and adversarial loss, the weights of each loss can be determined based on actual needs, and the weighted sum of the losses can be calculated to obtain the global loss. The formula for calculating the global loss is as follows:

[0189] loss = w1* Reconstruction_loss + w2 * LPIPS_loss + w3 * ID_loss +w4*G_loss

[0190] Where loss represents the global loss of the initial image processing model, w1 represents the weight of pixel reconstruction loss (which can be 20), w2 represents the weight of perceptual loss (which can be 20), w3 represents the weight of identity feature loss (which can be 5), and w4 represents the weight of adversarial loss (which can be 1).

[0191] After obtaining the global loss of the initial image processing model, the parameters in the initial image processing model are trained based on the global loss and the first training image set until the model converges, thus obtaining the trained model.

[0192] It's important to note that during the training of the initial image processing model, the initial feature fusion network and the initial discriminator network are trained in a generative adversarial manner. Specifically, the initial feature fusion network randomly samples from the latent space as input, and its output needs to mimic real samples in the training set as closely as possible. The initial discriminator network takes the output of the initial feature fusion network as input, and its goal is to distinguish the output of the initial feature fusion network from real samples as much as possible. The initial feature fusion network, in turn, tries to "deceive" the initial discriminator network as much as possible. The two networks compete against each other, constantly adjusting their parameters, ultimately generating a highly realistic image.

[0193] In practical applications, the target image processing model only needs to use the trained feature fusion network for feature fusion processing, without needing a discriminant network for authenticity judgment. Therefore, the target image processing model does not need to include a discriminant network. In other words, compared to the architecture of the initial image processing model, the target image processing model does not include a discriminant network.

[0194] Figure 11This is a schematic diagram of the target image processing model architecture. The target image processing model includes a first encoding network, a second encoding network, a mapping network, a feature fusion network, and a decoding network. The object template is sequentially passed through the first encoding network for feature compression, feature sampling, and layer-by-layer encoding to obtain multiple object attribute features with different feature sizes. The identity image is sequentially passed through the second encoding network and the mapping network for feature compression and nonlinear mapping to obtain object identity features. The object identity features are then input into each feature generation layer of the feature fusion network, and each object attribute feature is input into a feature generation layer with a matching feature size. For any feature generation layer, the feature size of the object identity features is transformed to obtain transformed identity features that match the feature size of the feature generation layer. The transformed identity features, object attribute features, and the output features of the previous feature generation layer are superimposed to obtain the initial object fusion features. Convolution is performed on the initial object fusion features to obtain the output features corresponding to the feature generation layer. The output features of the last feature generation layer in the feature fusion network are determined as the object fusion features. The object fusion features are then decoded by the decoding network to improve resolution, and the target object image is output.

[0195] In the above embodiments, by calculating the pixel reconstruction loss and perceptual loss between the training object template image and the target training object image, and the identity feature loss and adversarial loss between the training identity image and the target training object image, the training effect of the initial feature fusion network throughout the entire training process is comprehensively and objectively characterized. Based on this, the weights of the reconstruction loss, perceptual loss, identity feature loss, and adversarial loss can be flexibly adjusted according to actual needs. By increasing or decreasing the weight ratio of each loss in the global loss, the initial image processing network can be trained specifically to meet the application requirements of different image processing scenarios.

[0196] In one embodiment, the feature error norm between the training object template image and the target training object image is calculated to obtain the perceptual loss, including:

[0197] Training attribute features of the training object template image at different layers are extracted using a deep convolutional network, and target attribute features of the target training object image at different layers are extracted using a deep convolutional network; the sum of the Euclidean distances between the training attribute features and the target attribute features is determined as the perceptual loss.

[0198] The deep convolutional network can be a pre-trained attribute feature extraction network; please refer to [reference needed]. Figure 12 , Figure 12 This is a schematic diagram of a deep convolutional network architecture. Low-level features can represent low-level features such as lines and colors, while high-level features can represent high-level features such as components. It should be understood that... Figure 12This is only a schematic diagram of the architecture. The number of network layers included in a deep convolutional network can be 5, 8, 128, etc., and this embodiment does not limit this.

[0199] In this embodiment, a deep convolutional network is applied to both the training template image and the target training object image. Specifically, the deep convolutional network extracts training attribute features from the training template image at different layers, and extracts target attribute features from the target training object image at different layers. It should be understood that the feature extraction methods for the training template image and the target training object image are consistent, and correspondingly, the training attribute features and target attribute features are also matched.

[0200] After extracting the training attribute features of the training template image and the target attribute features of the target training object image, the computer device can measure the overall similarity between the training template image and the target training object image by calculating the Euclidean distance between the training attribute features and the target attribute features.

[0201] Taking the training attribute features of the training object template image as (fake_fea1, fake_fea2, fake_fea3, fake_fea4) and the target attribute features of the target training object image as (GT_fea1, GT_fea2, GT_fea3, GT_fea4) as an example, the calculation formula for perceptual loss (LPIPS_loss) is as follows:

[0202] LPIPS_loss = || fake_fea1 - GT_fea1 || + || fake_fea2 - GT_fea2|| + || fake_fea3 - GT_fea3 || + || fake_fea4- GT_fea4 ||

[0203] Where LPIPS_loss represents perceptual loss, fake_feax represents the x-th training attribute feature of the target training object image, GT_feax represents the x-th target attribute feature of the training object template image, and || represents performing Euclidean distance calculation to obtain the L2 norm.

[0204] In the above embodiments, training attribute features are used to characterize the low-level and high-level features of the training object template image at different layers, and target attribute features are used to characterize the low-level and high-level features of the target training object image at different layers, so as to fully compare the feature differences between the training object template image and the target training object image, thereby improving the accuracy and comprehensiveness of the perceptual loss obtained based on the sum of the Euclidean distances between the training attribute features and the target attribute features.

[0205] In one embodiment, the feature similarity between the training identity image and the target training object image is calculated to obtain the identity feature loss, including:

[0206] The training identity features of the training identity image are extracted using an object model, and the target identity features of the target training object image are extracted using the same object model. The feature similarity between the training identity features and the target identity features is calculated to obtain the identity feature loss.

[0207] Among them, the object model is a pre-trained identity feature extraction network used to extract identity features from images. The dimension of this feature is generally 1024.

[0208] In this embodiment, the object model is applied to both the training identity image and the target training object image. Specifically, the object model is used to extract the training identity features of the training identity image and the target identity features of the target training object image.

[0209] It should be understood that the feature extraction methods for the training identity image and the target training object image are consistent, and correspondingly, the training identity features and the target identity features are also matched, for example, both are 1024-dimensional.

[0210] After extracting the training identity features of the training identity image and the target identity features of the target training object image, the computer device can calculate the cosine similarity between the training identity features and the target identity features, and then obtain the identity feature loss based on the cosine similarity to measure the overall similarity between the training identity image and the target training object image.

[0211] The formula for calculating the identity feature loss is as follows:

[0212] ID_loss = 1–cosine_similarity(fake_id_features, source_id_features)

[0213] Here, ID_loss represents the identity feature loss, fake_id_features represents the training identity features, source_id_features represents the target identity features, and cosine_similarity() represents the cosine similarity operation. The closer the training identity features are to the target identity features, the larger the cosine similarity value and the smaller the identity feature loss.

[0214] Cosine similarity can be used The specific calculation formula is as follows:

[0215]

[0216] Among them, Ai B represents the feature in the training identity features. i This represents a feature within the target's identity characteristics.

[0217] In the above embodiments, the training identity features of the training identity image and the target identity features of the target training object image are extracted by the pre-trained object model. This improves the feature extraction speed and makes the training identity features and target identity features mutually exclusive, so as to facilitate the subsequent calculation of cosine similarity and improve the speed of obtaining identity feature loss.

[0218] In one specific embodiment, the method of this application can be applied to the scenario of ID photo creation. This application proposes a face-swapping application for template ID photos, supporting users to modify the identity information in any type of ID template image with the identity information of their own facial image, based on their own facial image.

[0219] In this scenario, the object template image is the document template image, the identity image is the user's face image, the target object image is the target document image, and further, the object attribute feature distribution is the document attribute feature distribution, the initial object attribute features are the initial document template attribute features, the object attribute features are the document template features, the initial object identity features are the initial face identity features, the object identity features are the user's face features, and the object fusion features are the user document fusion features.

[0220] In response to a user-triggered face-swapping command for a user's face image and an ID card template image, the computer device performs pre-processing for face-swapping. Specifically, it performs face detection on the ID card template image, obtains the face region of the ID card template image, and performs image registration on the face region. It also performs face detection on the user identity image, obtains the face region of the user identity image, and performs image registration on the face region. Then, it normalizes the resolution of the image-registered ID card template image and the image-registered user identity image.

[0221] Next, the computer device performs feature compression on the document template image through the first encoding network to extract attribute features, obtains the document attribute feature distribution, and selects features that conform to the normal distribution from the document attribute feature distribution as the initial document template attribute features. Then, the initial document template features are encoded layer by layer through alternating convolutional layers and downsampling layers to obtain multiple document template features with different feature sizes.

[0222] While acquiring the document template features, the user's face image is transformed through a second encoding network to obtain initial face identity features. The initial face identity features are then nonlinearly mapped to obtain the user's face features.

[0223] Then, the document template features and user face features are input into the trained feature fusion network. The input user face features are transformed by feature generation layers with different feature sizes in the feature fusion network to obtain transformed user face features that match the feature size of the feature generation layers. The transformed user face features are superimposed and convolved with the document template features to output intermediate fusion features. The output features of the last feature generation layer are determined as the user document fusion features.

[0224] Finally, the user's ID fusion features are upsampled using a decoding network to obtain the target ID image.

[0225] In this embodiment, the above steps can be integrated into a target image processing model and pre-deployed on a computer device. In this way, the computer device can directly call the target image processing model to process the user's face image and the document template image, output the target document image, shorten the image processing time, and improve the image processing efficiency.

[0226] The target image processing model can be divided into three training stages: encoding and decoding network (including the first encoding network and the decoding network), second encoding network, and feature fusion network.

[0227] In the first training phase, training samples and training labels are acquired. The training samples are input into the first initial encoding network to obtain the sample object attribute features. The sample object attribute features are then input into the initial decoding network to obtain the sample training results. The similarity loss is obtained by calculating the difference between the feature distribution of the sample object attribute features and the standard normal distribution. The reconstruction loss is obtained by calculating the feature norm of the sample training results and the training labels. The reconstruction loss and the similarity loss are superimposed to obtain the encoding and decoding loss. Based on the encoding and decoding loss, the first initial encoding network and the initial decoding network are trained simultaneously to obtain the first encoding network and the decoding network.

[0228] In the second training phase, training images and training feature labels are acquired. The training images are input into the second initial encoding network to obtain training identity features. Based on the feature loss of the training identity features and training feature labels, the network parameters in the second initial encoding network are optimized until the second initial encoding network converges, thus obtaining the second encoding network.

[0229] In the third training phase, the first encoding network, decoding network, second encoding network, initial feature fusion network, and initial discriminator network are concatenated to obtain the initial image processing model. Based on the acquired training object template image and training identity image, these images are input into the initial image processing model to obtain the target object image. The pixel reconstruction loss between the training object template image and the target training object image is calculated; the feature error norm between them is calculated to obtain the perceptual loss; the feature similarity between the training identity image and the target training object image is calculated to obtain the identity feature loss; the adversarial loss is obtained based on the initial discriminator network's judgment of the target training object image's authenticity; and the global loss of the initial image processing model is obtained based on the pixel reconstruction loss, perceptual loss, identity feature loss, and adversarial loss. The parameters of the initial feature fusion network and the initial discriminator network in the initial image processing model are then trained based on this global loss to obtain the trained feature fusion network. Finally, the first encoding network, decoding network, second encoding network, and feature fusion network are concatenated to obtain the target image processing model.

[0230] In the process of replacing facial information in ID photos in the above embodiments, facial features are compressed through the first and second encoding networks to obtain low-resolution ID template features and user facial features. This reduces the number of learning parameters for the feature fusion network, and the speed of generating user ID fusion features through feature fusion is also faster. Furthermore, decoding the generated user ID fusion features can ensure the quality of the target ID image obtained by face swapping. This ID photo face swapping method not only ensures the quality of face swapping but also improves the efficiency of face swapping, thus enhancing the user ID face swapping experience from multiple dimensions.

[0231] In one specific embodiment, the image processing method provided in this application can be applied to the production of film and television portraits. In the prior art, when actors are unable to perform professional actions, professional personnel can complete the filming of professional videos. Later, the image processing method provided in this application, which uses the actor's identity image to swap the professional personnel's face in the professional videos, replaces the identity information of the professional stunt double with the actor's identity information.

[0232] In this scenario, the object template image is any image in the object template video, the identity image is the actor's identity image, and the target object image is the corresponding image in the target video.

[0233] The method described in this application includes acquiring object template videos, inputting identity images, pre-processing (face detection and face region cropping), face swapping of each object template video in the object template video based on a target image processing model, and obtaining and displaying the target video.

[0234] For the template video acquisition stage, videos of professionals performing professional actions are filmed; for the identity image input stage, actor identity images representing actor identity information are obtained.

[0235] For the pre-processing stage, facial recognition is performed on the actor's identity image, the actor's face region is cropped, and image registration is performed to obtain the image-registered actor's identity image. For each object template image in the object template video, face detection is performed, the professional face region in each object template image is cropped, and image registration is performed on each professional face region to obtain the image-registered object template image. Finally, the resolution of the image-registered actor's identity image and the image-registered object template image are normalized.

[0236] For the video face-swapping process, for any image-registered object template image, the object template image and the actor's identity image are input into the target image processing model to obtain the corresponding target object image. The identity features of the target object image match the actor's identity image, and the expression features, angle features, and attribute features of the target object image match the object template image. This process is repeated iteratively to obtain the target object image corresponding to each image-registered object template image. Finally, all target object images are aggregated to obtain the target video.

[0237] The target image processing model is trained in stages: the first training stage is to pre-train the first encoding network and the decoding network; the second training stage is to pre-train the second encoding network; and the third training stage is to train the feature fusion network.

[0238] In the first training phase, training samples and training labels are acquired. The training samples are input into the first initial encoding network to obtain the sample object attribute features. The sample object attribute features are then input into the initial decoding network to obtain the sample training results. The similarity loss is obtained by calculating the difference between the feature distribution of the sample object attribute features and the standard normal distribution. The reconstruction loss is obtained by calculating the feature norm of the sample training results and the training labels. The reconstruction loss and the similarity loss are superimposed to obtain the encoding and decoding loss. Based on the encoding and decoding loss, the first initial encoding network and the initial decoding network are trained simultaneously to obtain the first encoding network and the decoding network.

[0239] In the second training phase, training images and training feature labels are acquired. The training images are input into the second initial encoding network to obtain training identity features. Based on the feature loss of the training identity features and training feature labels, the network parameters in the second initial encoding network are optimized until the second initial encoding network converges, thus obtaining the second encoding network.

[0240] In the third training phase, the first encoding network, decoding network, second encoding network, initial feature fusion network, and initial discriminator network are concatenated to obtain the initial image processing model. Based on the acquired training object template image and training identity image, these images are input into the initial image processing model to obtain the target object image. The pixel reconstruction loss between the training object template image and the target training object image is calculated; the feature error norm between them is calculated to obtain the perceptual loss; the feature similarity between the training identity image and the target training object image is calculated to obtain the identity feature loss; the adversarial loss is obtained based on the initial discriminator network's judgment of the target training object image's authenticity; and the global loss of the initial image processing model is obtained based on the pixel reconstruction loss, perceptual loss, identity feature loss, and adversarial loss. The parameters of the initial feature fusion network and the initial discriminator network in the initial image processing model are then trained based on this global loss to obtain the trained feature fusion network. Finally, the first encoding network, decoding network, second encoding network, and feature fusion network are concatenated to obtain the target image processing model.

[0241] In the above embodiment, during the video face-swapping process, facial features are compressed through a first encoding network and a second encoding network to obtain low-resolution object template images and actor identity images. This reduces the number of learning parameters for the feature fusion network, making it less prone to overfitting. While improving the generalization of the feature fusion network, it generates high-quality target object images, thereby obtaining high-quality target videos.

[0242] In one specific embodiment, the image processing method provided in this application can be applied to face replacement scenarios in videos of actors with criminal records. In the prior art, if a video cannot be played due to the actor's own reasons after filming, it needs to be reshot, wasting significant resources. Therefore, this application provides a method for face-swapping actors with criminal records, replacing their identity information with that of other actors, avoiding reshoots and saving filming costs.

[0243] The method described in this application includes obtaining template videos of actors with bad records, inputting identity images, pre-processing (face detection and face region cropping), face swapping of each object template video in the object template video based on a target image processing model, and obtaining and displaying the target video.

[0244] It should be noted that the method provided in this embodiment differs from the image processing method for face-swapping professional personnel videos based on actor identity images in the previous embodiments in that the processing objects are different, but the processing method is the same. The processing method can be found in the above embodiments and will not be repeated here.

[0245] In one specific embodiment, the image processing method provided in this application can be applied to virtual video face-swapping scenarios. In this scenario, the object template image is any user image in the user's live video, the identity image is a virtual image, and the target object image is a fun image obtained by performing a fun face-swapping on the user image.

[0246] During a user's live stream, in response to a user-triggered fun face-swapping command, the target image processing model is invoked to process the virtual image and the user image to obtain a fun image. Specifically, the facial features in the user image are replaced with facial features in the virtual image to synthesize the fun image, which is then displayed in the user's live stream.

[0247] The training process of the target image processing model is the same as the multi-stage training process in the aforementioned embodiments, including:

[0248] The training samples and training labels are input into the first initial encoding network to obtain the sample object attribute features. The sample object attribute features are then input into the initial decoding network to obtain the sample training results. The similarity loss is obtained by calculating the difference between the feature distribution of the sample object attribute features and the standard normal distribution. The reconstruction loss is obtained by calculating the feature norm of the sample training results and the training labels. The reconstruction loss and the similarity loss are superimposed to obtain the encoding and decoding loss. The first initial encoding network and the initial decoding network are trained simultaneously based on the encoding and decoding loss to obtain the first encoding network and the decoding network.

[0249] Acquire training images and training feature labels, input the training images into the second initial encoding network to obtain training identity features, optimize the network parameters in the second initial encoding network based on the feature loss of the training identity features and training feature labels, until the second initial encoding network converges to obtain the second encoding network.

[0250] The training object template image is input into the first encoding network to obtain the training object attribute features. The training identity image is input into the second encoding network to obtain the training object identity features. The training object attribute features and training object identity features are input into the initial feature fusion network to obtain the training object fusion features. The training object fusion features are input into the decoding network to obtain the corresponding target training object image. The pixel reconstruction loss between the training object template image and the target training object image is calculated. The training attribute features of the training object template image at different layers are extracted through a deep convolutional network, and the target attribute features of the target training object image at different layers are extracted through a deep convolutional network. The training attribute features and the target... The sum of Euclidean distances between attribute features is determined as the perceptual loss; training identity features are extracted from the training identity image through the object model, and target identity features are extracted from the target training object image through the object model; the feature similarity between the training identity features and the target identity features is calculated to obtain the identity feature loss; the adversarial loss is obtained based on the authenticity judgment result of the initial discriminator network on the target training object image; the global loss of the initial image processing model is obtained based on the pixel reconstruction loss, perceptual loss, identity feature loss, and adversarial loss; the parameters of the initial feature fusion network and the initial discriminator network in the initial image processing model are trained based on the global loss to obtain the target image processing model.

[0251] In the above embodiments, during the video face-swapping process, the virtual image and the user's face image are compressed by the target image processing model, and then feature fusion is performed. This can quickly realize the replacement of facial features in the user's face image, which can enhance the fun of live video while meeting the real-time requirements of image processing in live streaming scenarios.

[0252] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0253] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0254] In one embodiment, such as Figure 13 As shown, an image processing apparatus is provided, including: an image acquisition module 1302, a feature extraction module 1304, a feature fusion module 1306, and a feature decoding module 1308, wherein:

[0255] Image acquisition module 1302 is used to acquire object template image and identity image;

[0256] The feature extraction module 1304 is used to compress the object template image to obtain initial object attribute features, and to encode the initial object attribute features step by step to obtain multiple object attribute features with different feature sizes; and to compress the identity image to obtain object identity features.

[0257] The feature fusion module 1306 is used to input the object identity features and the object attribute features into the feature fusion network to obtain the object fusion features; the feature fusion network includes multiple feature generation layers with different feature sizes, and the object attribute features are used to input into the feature generation layer with matching feature sizes.

[0258] The feature decoding module 1308 is used to perform resolution-enhancing decoding on the object fusion features to obtain the corresponding target object image.

[0259] In one embodiment, the feature extraction module 1304 is further configured to:

[0260] The object template image is compressed using the first encoding network to extract attribute features, resulting in an object attribute feature distribution. Object attribute features that conform to a normal distribution are selected from the object attribute feature distribution as initial object attribute features.

[0261] In one embodiment, the feature extraction module 1304 is further configured to:

[0262] The identity image is transformed using a second coding network to obtain the initial object identity features;

[0263] The initial object identity features are nonlinearly mapped to obtain the object identity features.

[0264] In one embodiment, the feature fusion module 1306 is further configured to:

[0265] Input object attribute features of different feature sizes into a feature generation layer that matches the feature size;

[0266] Input the object's identity features into each feature generation layer;

[0267] For any feature generation layer, the feature size of the object identity feature is transformed through the feature generation layer to obtain the transformed identity feature that matches the feature size of the feature generation layer. The transformed identity feature, object attribute feature and the output feature of the previous feature generation layer are superimposed to obtain the initial object fusion feature. The initial object fusion feature is convolved to obtain the output feature corresponding to the feature generation layer.

[0268] The output features of the last feature generation layer in the feature fusion network are determined as the object fusion features.

[0269] In one embodiment, the feature decoding module 1308 is further configured to:

[0270] The object fusion features are upsampled by a decoding network to obtain the target object image; the resolution of the target object image matches the resolution of the object template image.

[0271] In one embodiment, the image processing apparatus further includes a preprocessing module and a normalization processing module, wherein:

[0272] The preprocessing module is used to perform object recognition on the object template image to obtain a first region of interest (ROI) of the object template image and perform image registration on the first ROI; and to perform object recognition on the identity image to obtain a second ROI of the identity image and perform image registration on the second ROI.

[0273] The normalization module is used to normalize the resolution of the object template image and the identity image after image registration.

[0274] In one embodiment, the object template image is any image in the object template video; the image processing apparatus further includes a feature acquisition module, a resolution processing module, and a video acquisition module, wherein:

[0275] The feature acquisition module is used to acquire the object attribute features of each object template image;

[0276] The resolution processing module is used to fuse the object identity features with the object attribute features of each object template video to obtain multiple object fusion features. The resolution of each object fusion feature is then enhanced by decoding to obtain the corresponding target object images.

[0277] The video acquisition module is used to obtain the corresponding target video based on the images of each target object.

[0278] In one embodiment, the object template image is an ID card template image; the target object image is a target ID card image, the identity features of the target ID card image are matched with the identity image, and the facial expression features, angle features, and background features of the target ID card image are matched with the ID card template image.

[0279] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0280] Based on the same inventive concept, this application also provides an image processing model training apparatus for implementing the image processing model training method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing model training method described above, and will not be repeated here.

[0281] In one exemplary embodiment, such as Figure 14 As shown, an image processing model training device is provided, including: an image set acquisition module 1402 and a model training module 1404, wherein:

[0282] The image set acquisition module 1402 is used to acquire a first training image set, which includes training object template images and training identity images.

[0283] The model training module 1404 is used to input the training object template image and the training identity image into the initial image processing model for training to obtain the corresponding target image processing model. The target image processing model includes a feature fusion network, which includes multiple feature generation layers with different feature sizes. The feature fusion network is used to obtain the training object fusion features based on the training identity features and training attribute features. The training identity features are obtained by feature compression of the training identity image, and the training attribute features are obtained by feature compression of the training object template image to obtain the initial object attribute features, and the initial object attribute features are encoded step by step. The training object attribute features are used to input into the feature generation layer with matching feature sizes.

[0284] The target image processing model is used to process the input object template image and identity image to obtain the corresponding target object image; the target object image is obtained by decoding the object fusion features corresponding to the object template image and identity image with improved resolution.

[0285] In one embodiment, the target image processing model further includes a first encoding network and a decoding network; the model training module 1404 is also used for:

[0286] Obtain a second training image set, which includes training samples and training labels;

[0287] The training samples are input into the first initial encoding network to obtain the sample object attribute features, and the sample object attribute features are input into the initial decoding network to obtain the sample training results;

[0288] The similarity loss is obtained by calculating the difference between the feature distribution of the sample object's attribute features and the standard normal distribution; the reconstruction loss is obtained by calculating the feature norm of the sample training results and the training labels.

[0289] The reconstruction loss and the similarity loss are superimposed to obtain the encoding and decoding loss. Based on the encoding and decoding loss, the first initial encoding network and the initial decoding network are trained simultaneously to obtain the first encoding network and the decoding network.

[0290] In one embodiment, the initial image processing model includes a first encoding network, a second encoding network, a decoding network, an initial feature fusion network, and an initial discriminant network; the model training module 1404 is further configured to:

[0291] The training object template image is input into the first encoding network to obtain the training object attribute features. The training identity image is input into the second encoding network to obtain the training object identity features. The training object attribute features and the training object identity features are input into the initial feature fusion network to obtain the training object fusion features. The training object fusion features are input into the decoding network to obtain the corresponding target training object image.

[0292] Calculate the pixel reconstruction loss between the training object template image and the target training object image; calculate the feature error norm between the training object template image and the target training object image to obtain the perceptual loss; calculate the feature similarity between the training identity image and the target training object image to obtain the identity feature loss; and obtain the adversarial loss based on the authenticity judgment result of the initial discriminant network on the target training object image.

[0293] The global loss of the initial image processing model is obtained based on the pixel reconstruction loss, the perception loss, the identity feature loss, and the adversarial loss. The parameters of the initial feature fusion network and the initial discriminant network in the initial image processing model are trained based on the global loss to obtain the target image processing model.

[0294] In one embodiment, the model training module 1404 is further configured to:

[0295] The training attribute features of the training object template image at different layers are extracted by a deep convolutional network, and the target attribute features of the target training object image at different layers are extracted by the deep convolutional network.

[0296] The sum of the Euclidean distances between the training attribute features and the target attribute features is determined as the perceptual loss.

[0297] In one embodiment, the model training module 1404 is further configured to:

[0298] The training identity features of the training identity image are extracted using the object model, and the target identity features of the target training object image are extracted using the object model.

[0299] Calculate the feature similarity between the trained identity features and the target identity features to obtain the identity feature loss.

[0300] Each module in the aforementioned image processing model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0301] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 15 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores training image sets, initial image processing models, target image processing models, and other data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an image processing method or an image processing model training method.

[0302] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 16 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method or an image processing model training method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0303] Those skilled in the art will understand that Figure 15 , Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0304] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0305] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0306] In one embodiment, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0307] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0308] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0309] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0310] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Retrieve the object template image and identity image; The object template image is subjected to feature compression to obtain initial object attribute features, and the initial object attribute features are progressively encoded to obtain multiple object attribute features with different feature sizes; the identity image is subjected to feature compression to obtain object identity features; The object identity features and each of the object attribute features are input into a feature fusion network to obtain object fusion features; the feature fusion network includes multiple feature generation layers with different feature sizes, and the object attribute features are used to be input into the feature generation layer with matching feature sizes. The object fusion features are then subjected to resolution-enhanced decoding to obtain the corresponding target object image.

2. The method according to claim 1, characterized in that, The step of performing feature compression on the object template image to obtain initial object attribute features includes: The object template image is subjected to feature compression by the first encoding network to extract attribute features, thereby obtaining the object attribute feature distribution. Object attribute features that conform to the normal distribution are selected from the object attribute feature distribution as the initial object attribute features.

3. The method according to claim 1, characterized in that, The step of compressing the identity image to obtain the object's identity features includes: The identity image is transformed using a second coding network to obtain initial object identity features; The initial object identity features are nonlinearly mapped to obtain the object identity features.

4. The method according to claim 1, characterized in that, The step of inputting the object identity features and each of the object attribute features into a feature fusion network to obtain object fusion features includes: Input object attribute features of different feature sizes into a feature generation layer that matches the feature size; The object identity features are input into each of the feature generation layers; For any feature generation layer, the object identity feature is transformed by the feature generation layer to obtain a transformed identity feature that matches the feature size of the feature generation layer. The transformed identity feature, the object attribute feature and the output feature of the previous feature generation layer are superimposed to obtain an initial object fusion feature. The initial object fusion feature is then convolved to obtain the output feature corresponding to the feature generation layer. The output feature of the last feature generation layer in the feature fusion network is determined as the object fusion feature.

5. The method according to claim 1, characterized in that, The step of performing resolution-enhancing decoding on the object fusion features to obtain the corresponding target object image includes: The object fusion features are upsampled by a decoding network to obtain the target object image; the resolution of the target object image matches the resolution of the object template image.

6. The method according to claim 1, characterized in that, Before performing feature compression on the object template image, the method further includes: The object template image is subjected to object recognition to obtain a first region of interest (ROI) of the object template image, and the first ROI is image registered; and the identity image is subjected to object recognition to obtain a second ROI of the identity image, and the second ROI is image registered. The resolutions of the image-registered object template image and the image-registered identity image are normalized.

7. The method according to claim 1, characterized in that, The object template image is any image in the object template video; the method further includes: Obtain the object attribute features of each of the object template images; The object identity features are fused with the object attribute features of each object template video to obtain multiple object fusion features. The object fusion features are then decoded with increased resolution to obtain the corresponding target object images. Based on the images of each target object, the corresponding target video is obtained.

8. The method according to claim 1, characterized in that, The object template image is an ID card template image; the target object image is a target ID card image, the identity features of the target ID card image are matched with the identity image, and the facial expression features, angle features, and background features of the target ID card image are matched with the ID card template image.

9. A method for training an image processing model, characterized in that, The method includes: Obtain a first training image set, which includes training object template images and training identity images; The first training image set is input into the initial image processing model for training to obtain the corresponding target image processing model. The target image processing model includes a feature fusion network, which includes multiple feature generation layers with different feature sizes. The feature fusion network is used to obtain training object fusion features based on training object identity features and training object attribute features. The training object identity features are obtained by feature compression of the training identity image. The training object attribute features are obtained by feature compression of the training object template image to obtain initial training object attribute features, and then progressively encoding the initial training object attribute features. The training object attribute features are used to input into the feature generation layer with matching feature sizes. The target image processing model is used to process the input object template image and identity image to obtain the corresponding target object image; the target object image is obtained by decoding the object fusion features corresponding to the object template image and the identity image with increased resolution.

10. The method according to claim 9, characterized in that, The target image processing model further includes a first encoding network and a decoding network; prior to acquiring the first training image set, the method further includes: Obtain a second training image set, which includes training samples and training labels; The training samples are input into the first initial encoding network to obtain the sample object attribute features, and the sample object attribute features are input into the initial decoding network to obtain the sample training results; The similarity loss is obtained by calculating the difference between the feature distribution of the sample object's attribute features and the standard normal distribution; the reconstruction loss is obtained by calculating the feature norm of the sample training results and the training labels. The reconstruction loss and the similarity loss are superimposed to obtain the encoding and decoding loss. Based on the encoding and decoding loss, the first initial encoding network and the initial decoding network are trained simultaneously to obtain the first encoding network and the decoding network.

11. The method according to claim 9, characterized in that, The initial image processing model includes a first encoding network, a second encoding network, a decoding network, an initial feature fusion network, and an initial discrimination network; The step of inputting the first training image set into the initial image processing model for training to obtain the corresponding target image processing model includes: The training object template image is input into the first encoding network to obtain the training object attribute features. The training identity image is input into the second encoding network to obtain the training object identity features. The training object attribute features and the training object identity features are input into the initial feature fusion network to obtain the training object fusion features. The training object fusion features are input into the decoding network to obtain the corresponding target training object image. Calculate the pixel reconstruction loss between the training object template image and the target training object image; calculate the feature error norm between the training object template image and the target training object image to obtain the perceptual loss; calculate the feature similarity between the training identity image and the target training object image to obtain the identity feature loss; and obtain the adversarial loss based on the authenticity judgment result of the initial discriminant network on the target training object image. The global loss of the initial image processing model is obtained based on the pixel reconstruction loss, the perception loss, the identity feature loss, and the adversarial loss. The parameters of the initial feature fusion network and the initial discriminant network in the initial image processing model are trained based on the global loss to obtain the target image processing model.

12. The method according to claim 11, characterized in that, The step of calculating the feature error norm between the training object template image and the target training object image to obtain the perceptual loss includes: The training attribute features of the training object template image at different layers are extracted by a deep convolutional network, and the target attribute features of the target training object image at different layers are extracted by the deep convolutional network. The sum of the Euclidean distances between the training attribute features and the target attribute features is determined as the perceptual loss.

13. The method according to claim 11, characterized in that, The step of calculating the feature similarity between the training identity image and the target training object image to obtain the identity feature loss includes: The training identity features of the training identity image are extracted using the object model, and the target identity features of the target training object image are extracted using the object model. Calculate the feature similarity between the trained identity features and the target identity features to obtain the identity feature loss.

14. An image processing apparatus, characterized in that, The device includes: The image acquisition module is used to acquire object template images and identity images; The feature extraction module is used to compress the object template image to obtain initial object attribute features, and to encode the initial object attribute features step by step to obtain multiple object attribute features with different feature sizes; and to compress the identity image to obtain object identity features. The feature fusion module is used to input the object identity features and each of the object attribute features into the feature fusion network to obtain object fusion features; the feature fusion network includes multiple feature generation layers with different feature sizes, and the object attribute features are used to input into the feature generation layer with matching feature sizes; The feature decoding module is used to perform resolution-enhanced decoding processing on the fused features of the object to obtain the corresponding target object image.

15. An image processing model training device, characterized in that, The device includes: An image set acquisition module is used to acquire a first training image set, which includes training object template images and training identity images. The model training module is used to input the training object template image and the training identity image into an initial image processing model for training to obtain a corresponding target image processing model. The target image processing model includes a feature fusion network, which includes multiple feature generation layers with different feature sizes. The feature fusion network is used to obtain training object fusion features based on training identity features and training attribute features. The training identity features are obtained by feature compression of the training identity image. The training attribute features are obtained by feature compression of the training object template image to obtain initial object attribute features, and then progressively encoding the initial object attribute features. The training object attribute features are used to input into the feature generation layer with matching feature sizes. The target image processing model is used to process the input object template image and identity image to obtain the corresponding target object image; the target object image is obtained by decoding the object fusion features corresponding to the object template image and the identity image with increased resolution.

16. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 13.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is processed and executed, it implements the steps of the method according to any one of claims 1 to 13.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is processed and executed, it implements the steps of the method according to any one of claims 1 to 13.

Images