Image processing method and device and face image generation method and device

Abstract

The invention discloses an image processing method and device and a facial image generation method and device, and the processing method comprises the steps: detecting a target local position in a target image, and determining the target local position information of to-be-replaced target local information in the target image, shielding residual features of the target image of the target local information; according to detection of a reference local position in the reference image, inputting the determined reference local image in the reference image into a first network model for feature extraction, and determining a reference local feature; inputting the target local position information and the reference local feature into a second network model for feature integration, and generating an alignment feature for aligning the reference local feature with the target local position information; inputting the alignment feature and the residual feature of the target image into a third network model, and generating a replacement image for replacing the target local feature in the target image with the reference local feature; therefore, the naturalness of the generated image is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and more particularly to an image processing method and device. The present application also relates to a facial image generation method and device, as well as an image processing system, a computer storage medium, and an electronic device. Background Art

[0002] The rise and development of AIGC (Artificial Intelligence Generated Content) is the inevitable result of the rapid development of artificial intelligence technology and the growing prosperity of the content industry. The so-called AIGC refers to content generated by artificial intelligence systems, such as text, images, audio or video. It can quickly generate a large amount of content, thus saving time and resources. AIGC is mainly based on machine learning, especially technologies in the field of deep learning, such as natural language processing and computer vision. These technologies enable AI to autonomously generate various forms of content, such as text, images, audio and video. The current main application scenarios of AIGC can include automatic writing, image generation, audio creation and video production. With the continuous growth of digital content consumption, AIGC is increasingly widely used in industries such as media, advertising, design, and education. It can not only quickly generate a large amount of content, but also customize it according to user needs to improve the attractiveness and interactivity of the content. Summary of the invention

[0003] The present application provides an image processing method to solve the problem in the prior art that the generation of an image can only be based on the replacement of the original image from a coarse granularity, resulting in low controllability and freedom of image processing.

[0004] In view of this, the present application provides an image processing method, comprising:

[0005] Detecting the local position of the target in the target image, determining the target local position information of the target local information to be replaced in the target image, and shielding the remaining features of the target image of the target local information;

[0006] According to the detection of the reference local position in the reference image, the reference local image in the reference image is input into the first network model for feature extraction to determine the reference local feature;

[0007] Inputting the target local position information and the reference local feature into a second network model for feature integration, and generating an alignment feature for aligning the reference local feature with the target local position information;

[0008] The alignment features and the remaining features of the target image are input into a third network model to generate a replacement image in which the target local features in the target image are replaced with reference local features.

[0009] In some embodiments, the detecting the local position of the target in the target image, determining the target local position information of the target local information in the target image, and shielding the target image remaining features of the target local information include:

[0010] Determining position information of the local target information in the target image based on detection of the local target position in the target image;

[0011] generating a target partial image including the target partial information according to the target partial information corresponding to the position information;

[0012] According to the target partial image, a target partial position in the target image is masked to generate a target partial masked image;

[0013] The target local masked image is input into the fourth network model to extract the remaining features of the target image.

[0014] In some embodiments, the performing reference local position detection on the reference image to determine the reference local image in the reference image includes:

[0015] Determine reference local information corresponding to different reference local positions in the same reference image based on detection of different reference local positions in the same reference image; or determine reference local information corresponding to different reference local positions in different reference images based on detection of different reference local positions in multiple different reference images respectively;

[0016] According to the reference local position, image information other than the reference local information in the reference image is shielded to determine a plurality of reference local images.

[0017] In some embodiments, inputting the reference local image into the first network model to extract features and determine the reference local features includes:

[0018] The multiple reference local images are input into the first network model to extract local features, and the reference local features corresponding to the reference local images are determined.

[0019] In some embodiments, the inputting the target local position information and the reference local feature into a second network model for feature integration to generate an alignment feature for aligning the reference local feature with the target local position information includes:

[0020] Inputting the reference local features into the second network model to perform latent space feature splicing to generate a local feature splicing image;

[0021] The local feature stitching image is subjected to convolution processing, and the local stitching features in the local feature stitching image are aligned with the target position information to generate the alignment features.

[0022] In some embodiments, inputting the alignment features and the target image remaining features into a third network model to generate a replacement image in which the target local features in the target image are replaced with reference local features includes:

[0023] Inputting the alignment features and the target remaining features into the third network model to perform feature addition, and determining a replacement image that fuses the alignment features into the target image;

[0024] The replacement image is subjected to inverse diffusion to generate a target replacement image.

[0025] The present application also provides an image processing device, comprising:

[0026] A first determination unit is used to detect a local position of a target in a target image, determine target local position information of target local information in the target image, and shield remaining features of the target image of the target local information;

[0027] A second determining unit is used to input the determined reference local image in the reference image into the first network model for feature extraction to determine the reference local feature according to the detection of the reference local position in the reference image;

[0028] A first generating unit is used to input the target local position information and the reference local feature into a second network model for feature integration, and generate an alignment feature for aligning the reference local feature with the target local position information;

[0029] The second generating unit is used to input the alignment features and the remaining features of the target image into a third network model to generate a replacement image in which the target local features in the target image are replaced with reference local features.

[0030] The present application also provides a facial image generation method, comprising:

[0031] Detecting a local position of a target face in a target face image, determining target face local position information of the target face local information in the target face image, and shielding remaining features of the target face image of the target face local information;

[0032] According to the detection of the reference facial local position in the facial reference image, the reference facial local image in the facial reference image is input into the first network model for feature extraction to determine the reference facial local features;

[0033] Inputting the target facial local position information and the reference facial local features into a second network model for feature integration, and generating an alignment feature for aligning the reference facial local features with the target facial local position information;

[0034] The alignment features and the remaining features of the target facial image are input into a third network model to generate a replacement image in which the target facial local features in the target facial image are replaced with the reference facial local features.

[0035] The present application also provides a facial image generating device, comprising:

[0036] A first determining unit is used to detect a local position of a target face in a target face image, determine target face local position information of the target face local information in the target face image, and shield remaining features of the target face image of the target face local information;

[0037] A second determining unit is used to input the determined reference facial local image in the facial reference image into the first network model for feature extraction to determine the reference facial local features based on the detection of the reference facial local position in the facial reference image;

[0038] A first generating unit is used to input the target facial local position information and the reference facial local features into a second network model for feature integration, and generate an alignment feature for aligning the reference facial local features with the target facial local position information;

[0039] The second generating unit is used to input the alignment features and the remaining features of the target facial image into a third network model to generate a replacement image in which the target facial local features in the target facial image are replaced with the reference facial local features.

[0040] The present application also provides an image processing system, including: a detection module, a first network model, a second network model, a third network model, and a fourth network model;

[0041] The detection module is used to perform target local position detection on the input target image, determine the target local position information of the target local information to be replaced in the target image, and the remaining features of the target image after the target local information is masked; and to perform reference local position detection on the input reference image, and determine the reference local image in the reference image;

[0042] The first network model is used to input the reference local image into the first network model to extract features and determine reference local features;

[0043] The second network model is used to input the target local position information and the reference local feature into the second network model for feature integration, and generate an alignment feature for aligning the reference local feature with the target local position information;

[0044] The third network model is used to input the alignment features and the remaining features of the target image into the third network model to generate a replacement image in which the target local features in the target image are replaced with reference local features;

[0045] The fourth network module is used to extract the remaining features of the target image.

[0046] The present application also provides a computer storage medium, characterized in that it includes a computer program, and when the computer program is run on an electronic device, the electronic device executes the image processing method as described above, and / or executes the facial image generation method as described above.

[0047] The present application also provides an electronic device, including:

[0048] processor;

[0049] The memory is used to store a program for processing data generated by the electronic device. When the program is read and executed by the processor, the program executes the above-mentioned image processing method and / or the above-mentioned facial image generation method.

[0050] Compared with the prior art, this application has the following advantages:

[0051] An image processing method embodiment provided in the present application is mainly explained by taking facial images as an example. In fact, the target image and the reference image can be images of the same type. Of course, they can also be images of different types according to different image generation requirement scenarios. In this embodiment, the new facial image is generated by partial replacement of facial organs as an example. In scenes such as buildings and environments, the above method can still be used to replace local image elements to achieve fine-grained image generation, so that image generation no longer depends on the replacement of the overall synchronous movement mode. It can not only improve the naturalness of the generated image, but also has the advantage of fine-grained high controllability, and can integrate layout information from different reference images into the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of an image processing method provided by the present application;

[0053] Figure 2 It is a structural schematic diagram of an image processing device provided by the present application;

[0054] Figure 3 is a flow chart of a facial image generation method provided by the present application;

[0055] Figure 4 It is a schematic diagram of the execution process architecture of a facial image generation method provided by the present application;

[0056] Figure 5 It is a structural schematic diagram of a facial image generating device provided by the present application;

[0057] Figure 6 is a structural schematic diagram of an image processing system embodiment provided by the present application;

[0058] Figure 7 It is a structural schematic diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0059] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0060] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The descriptions used in this application and the appended claims, such as "a", "first", and "second", are not limitations on quantity or sequence, but are used to distinguish the same type of information from each other.

[0061] Based on the above background technology, it can be known that the inventive concept of the present application is derived from the technical defects or technical deficiencies in the field of image generation. Specifically, in the existing facial image generation scenario, at least the following two methods are usually used, namely: the first is to achieve it through overall facial replacement, and the other is to achieve it through facial mapping. The former is a coarse-grained implementation method, which replaces the facial contour as a whole to generate a new facial image. This method has low controllability and freedom for facial generation and is limited by the overall facial contour and content. The latter relies on a large amount of data for training, and the bias of the training data set may cause bias in the generated facial contour image. For example, if the data set mainly contains people of different genders or ages, the generated facial contour image may also be biased towards these features and lack diversity. In addition, the generated face is not real or natural in terms of details, expressions, light and shadow, etc. In short, the controllability of the image generated by the prior art is low.

[0062] In view of this, the present application provides an image processing method, the purpose of which is to improve the controllability of the image generation process, so that the generated image has more authenticity and freedom. Figure 1 As shown, Figure 1 is a flowchart of an image processing method provided by the present application, the method comprising:

[0063] Step S101: detecting a target local position in a target image, determining target local position information of target local information to be replaced in the target image, and shielding remaining features of the target image of the target local information;

[0064] Step S102: Based on the detection of the reference local position in the reference image, the reference local image in the reference image is input into the first network model for feature extraction to determine the reference local feature;

[0065] Step S103: inputting the target local position information and the reference local feature into a second network model for feature integration, and generating an alignment feature for aligning the reference local feature with the target local position information;

[0066] Step S104: inputting the alignment features and the remaining features of the target image into a third network model to generate a replacement image in which the target local features in the target image are replaced with reference local features.

[0067] Before describing the above steps S101 to S104 in detail, the related technical terms are explained as follows:

[0068] AIGC (AI-Generated Content) refers to the use of artificial intelligence to generate content. In a narrow sense, AIGC refers to the production method of automatically generating content using AI, such as automatic writing and automatic design. In a broad sense, AIGC refers to AI technology that has the ability to generate and create like humans. It can autonomously generate and create new texts, images, music, videos, 3D interactive content and other forms of content and data based on training data and generation algorithm models.

[0069] Image Generation (AI Image Generation), Artificial Intelligence (AI) can be used to generate images of works by non-human artists.

[0070] Video Generation (AI Video Generation),AIGC has been used for video clip processing to generate trailers and promotional videos.

[0071] Diffusion Model: It is a generative model based on probability theory. In computer vision, it trains neural networks by learning the inverse diffusion process to denoise Gaussian noise images and generate high-definition images.

[0072] U-Net network structure: It is a typical U-shaped structure that includes downsampling modules and upsampling modules. It includes two main parts: encoder (downsampling) and decoder (upsampling), which are connected by skip connections.

[0073] Facial local fusion: The information of different facial organs is fused together to create a new facial contour image.

[0074] Visual Grouding: Input text and an image, and output the location of the object specifically referred to by the text in the image.

[0075] VAE: Variational Autoencoder, a self-encoding network based on variational inference.

[0076] CLIP: Contrastive Language-Image Pre-Training, or CLIP for short, is a pre-trained neural network model for matching images and texts released by OpenAI in early 2021. It aims to embed images and texts into a shared semantic space through contrastive learning, that is, to match images and texts. It can achieve cross-modal understanding and the processing of multiple tasks, such as image classification, text classification, image retrieval, etc.

[0077] Open Set Delector: is a special type of object detector designed to handle scenes containing objects of known and unknown categories. It can detect and recognize not only the known object categories in the training set, but also those objects that do not belong to any known category, i.e., unknown category objects.

[0078] Mask-based Fusion Module: usually refers to a module that uses mask information for data fusion. Masks are often used in image processing to identify specific areas or objects in an image so that these areas can be processed or analyzed separately. In the fusion module, masks can be used to distinguish related parts of data from different sensors or different time points, thereby achieving more accurate data integration. In image processing scenarios: it can be used to merge specific areas in two or more images into one image while retaining useful information in each image. For example, in medical imaging, images of different modalities (such as CT and MRI) can be fused so that doctors can have a more comprehensive understanding of the patient's condition. In computer vision scenarios: it can be used for tasks such as target detection, image segmentation, and scene understanding. By generating a mask of the target object, the target object can be accurately extracted and fused. In deep learning scenarios: it can be used to build complex neural network models to achieve tasks such as image fusion, feature extraction, and target tracking. For example, in image fusion tasks, deep learning models can be used to generate high-quality masks and fuse multiple images based on these masks.

[0079] Addition-based Injection Module: is a structure commonly used in deep learning models, especially when dealing with visual tasks. The main function of this module is to effectively integrate features from different sources into the existing network. This module injects new features into a specific layer of the network through a simple addition operation. This simple addition operation makes information fusion more efficient and helps to preserve the information of the original features. It promotes the effective fusion of features through simple addition operations (additive multi-reference image fusion), which is suitable for a variety of deep learning tasks, especially in image processing and multimodal learning. Using this module, the performance of the model and the efficiency of information utilization can be improved. When designing a deep learning architecture, introducing such a module can help better handle complex data structures and feature mappings.

[0080] Regarding step S101: detecting the target local position in the target image, determining the target local position information of the target local information to be replaced in the target image, and shielding the remaining features of the target image of the target local information.

[0081] The purpose of step S101 is to obtain the position information of the part to be replaced in the target image, as well as other information after the replacement part is shielded. The specific implementation process may include:

[0082] Step S101-11: determining position information of the local target information in the target image based on detection of the local target position in the target image;

[0083] Step S101-12: generating a target partial image including the target partial information according to the target partial information corresponding to the position information;

[0084] Step S101 - 13: shielding a target local position in the target image according to the target local image to generate a target local shielded image;

[0085] Step S101-14: input the target local masked image into the fourth network model to extract the remaining features of the target image.

[0086] In this embodiment, the so-called target image can be understood as a reference image, and the image elements in the target image are replaced based on the target image, that is, the image elements are replaced with the same type of image elements in other images. For example, in the facial image processing scenario, the target image can be a facial image. In this embodiment, the facial image can include facial information and background information, for example, a frontal image of a human head including a human face; of course, it can also be other images with facial information. The target image can also include other information of a human body or other objects, such as the neck, torso, etc. of a human body or other animals, or a target image with a background image including facial information. The background can be other information other than the face, such as hair, scenery, etc., and can of course include images with occlusion on the face, such as images of people wearing glasses, images with headwear, earrings and other decorations, and images with other decorations on the face. The local image can be an organ image in a facial image, such as a nose, eyes, mouth and other facial organ images. Of course, the target image can also be a facial image of an animal, or other images, such as a landscape image, a building image, etc. In this embodiment, only the face is used as an example for illustration, and the purpose is to replace the organ image in the facial image with the same organ image in other images, so as to realize the generation of the facial image, that is, to realize the generation of the facial image by local replacement.

[0087] The detection of the target local position in step S101-11 may be performed by using Visual Grouding and Open Set Delector to detect the target local position in the target image, that is, to detect the position information of the facial region where the organ to be replaced is located in the target facial image.

[0088] The step S101 - 12 is to extract the local information and generate a target local image based on the detected position information.

[0089] The step S101-13 is to occlude the local position in the target image according to the target local image, thereby shielding the local position information in the target image and forming a target local shielded image. At this time, the local position in the target image only has other image pixel information left due to shielding. The fourth network model in step S101-14, which can be a VAE model, is used to extract the remaining image features of the target image.

[0090] In the following description, this embodiment is mainly described by taking facial images as an example, but the actual processing process is not limited to facial images.

[0091] Regarding step S102: based on the detection of the reference local position in the reference image, the reference local image in the reference image is input into the first network model for feature extraction to determine the reference local feature.

[0092] In this embodiment, the reference image is an image used to extract reference local information, and the reference local information is information used to replace the local information extracted from the target image. For example, in a face image, the target local information to be replaced in the target image is the eyes, nose and mouth, and the reference local image in the reference image is also the eyes, nose and mouth. Of course, it is not limited to the image organ replacement in the above example. The reference image may be the same as the target image, or may include a reference image of a face with decorations, and is not limited to facial features information. It may be non-facial features information, facial features information, or a combination of facial features information and other information as the reference local information.

[0093] It can be understood that the reference local position detection of the reference image can be performed on the positions of different reference local information in an image, for example: detecting the positions of the eyes, nose and mouth based on facial image A, or different reference local position information can be detected in multiple images, for example: detecting the position of the eyes based on facial image A, detecting the position of the nose based on facial image B, detecting the position of the mouth based on facial image C, etc.

[0094] With the continuous development of artificial intelligence and AIGC technology, both the detection of the target local position in the target image and the detection of the reference layout position in the reference image can be detected by text and image. The detection adopts Visual Grouding and Open Set Delector. Of course, other methods that can only detect the position of the image can also be adopted. Among them, the text can be the required text information input into the network model, or it can be voice information. The network model can convert the voice information into recognizable text information, and then determine the area that needs to be detected from the target image and / or the reference image, and determine the target local position information and / or reference local position information. Of course, this is only an implementation explanation when text information is input. In fact, the model can directly detect the facial image according to the training, so as to determine the position information of the facial features.

[0095] Based on the above, the specific implementation process of step S102 may include:

[0096] Step S102-11: determining reference local information corresponding to different reference local positions in the same reference image based on detection of different reference local positions in the same reference image; or determining reference local information corresponding to different reference local positions in different reference images based on detection of different reference local positions in multiple different reference images respectively;

[0097] Step S102-12: According to the reference local position, the remaining image information except the reference local information in the reference image is shielded to determine a plurality of reference local images.

[0098] In the description process of taking facial images as an example in this embodiment, the specific implementation process of step S102-11 may include:

[0099] Steps S102-111: Determine reference organ information corresponding to the position of the reference organ in the same facial reference image based on position detection of any reference organ or multiple reference organs of the facial features in the same facial reference image; or, determine reference organ information corresponding to each detection position in each facial reference image by performing position detection on different facial reference organs in multiple different facial reference images respectively.

[0100] The specific implementation process of step S102-12 may include:

[0101] Step S102-121: shielding the remaining facial image information except the reference organ information in the facial reference image according to the reference organ position, and determining the reference organ images respectively;

[0102] Step S102-122: extracting reference organ features according to the reference organ image.

[0103] The purpose of the steps S102-121 is to retain the reference organ information corresponding to the reference organ position, and to block or shield the image information of the remaining position areas in the reference image. For example, when the reference image is the same facial reference image a, the position of the eyes is determined from image a, and the image information of the eye position is retained, and the image information of the remaining positions is blocked; the position of the nose is determined from image a, and the image information of the nose position is retained, and the image information of the remaining positions is blocked; the position of the mouth is determined from image a, and the image information of the mouth position is retained, and the image information of the remaining positions is blocked. When the reference image is a plurality of facial reference images a, b, and c, the position of the eyes can be determined from image a, and the image information of the eye position can be retained, and the image information of the remaining positions except the eye position in image a can be blocked or shielded; the position of the nose can be determined from image b, and the image information of the nose position can be retained, and the image information of the remaining positions except the nose position in image b can be blocked or shielded; the position of the mouth can be determined from image c, and the image information of the mouth position can be retained, and the image information of the remaining positions except the mouth in image c can be blocked or shielded; thereby obtaining multiple reference organ images of the eyes, nose, mouth, etc., respectively.

[0104] The steps S102-122 may specifically be inputting a plurality of reference organ images into the first network model to extract reference organ features, thereby determining reference organ features. In this embodiment, the first network model may use a CLIP network model to extract reference organ features.

[0105] Regarding step S103: inputting the target local position information and the reference local feature into a second network model for feature integration, and generating an alignment feature for aligning the reference local feature with the target local position information.

[0106] The purpose of step S103 is to perform feature integration based on the target local position information determined in step S101 and the reference local features determined in step S102, so that the reference local features can be aligned with the target local position. The specific implementation process may include:

[0107] Step S103-11: inputting the reference local features into the second network model to perform latent space feature stitching to generate a local feature stitching image;

[0108] Step S103 - 12 : The local feature stitching image is subjected to convolution processing, and the local stitching features in the local feature stitching image are aligned with the target position information to generate the alignment features.

[0109] The purpose of step S103-11 and step S103-12 is to splice the extracted multiple reference local features through the second network model, and align the spliced ​​features with the target position, that is, to ensure that the generated replaced image can ensure that the reference local feature corresponds to the target local position of the target image. For example, when the reference local feature is an eye, the image information of the eye can replace the image information of the eye position area in the target image in the generated image, that is, to ensure the position accuracy and adaptability of the facial features after the replacement.

[0110] In this embodiment, the image features in multiple reference local images are fused separately. To ensure a smooth transition of feature fusion, the second network model adopts an addition-based multi-reference image fusion method to replace the local information in the image from a fine-grained level, and then generates an information image based on the local information. Therefore, the second network model is an Addition-based Injection Module, which fuses the image features in multiple reference local images in the latent space to generate a local feature spliced ​​image, and then aligns the local feature spliced ​​image with the local position information in the target image, that is, each reference local information in multiple reference local images is reflected in one image, and these reference local information correspond to the target local position information in the target image, for example: the reference local image of the eye, the reference local image of the nose, and the reference local image of the mouth are spliced ​​to form an image including eyes, nose, and mouth, and then the image is aligned with the target positions of the corresponding local positions of the eyes, nose, and mouth in the target image, and the organ image (local feature spliced ​​image) including the eyes, nose, and mouth is convolved. For example, a 1×1 convolution module can be used for convolution operation, and then the addition method is used for feature fusion, and the fused features are aligned with the target local position information in the channel dimension.

[0111] Regarding step S104: inputting the alignment features and the remaining features of the target image into a third network model to generate a replacement image in which the target local features in the target image are replaced with reference local features.

[0112] In this embodiment, the purpose is to replace the facial features information extracted from the reference image with the facial features information in the target image, so as to generate a new facial image. In step S102, the reference local features of the reference image are extracted, and in step S103, these reference local features are spliced ​​and aligned with the local position information in the target image. Step S104 needs to fuse the alignment features with the remaining features of the target image, that is, integrate the alignment features into the target image, so that the facial features information in the reference image is reflected in the facial features position information in the target image, so as to achieve the replacement of the facial features information. The replaced local information can be set according to the needs, which can be one or more of the facial eyes, nose, mouth, etc. Of course, the local information can also be replaced in other images, such as buildings, design drawings, etc., so as to improve the controllability of image generation from a fine-grained level, so that the generated image can be natural and highly adaptable, and the degree of freedom in the generation process is also correspondingly improved.

[0113] The third network model in step S104 may adopt a diffusion model, and the specific implementation process may include:

[0114] Step S104-11: inputting the alignment features and the target remaining features into the third network model to perform feature addition, and determining a replacement image that fuses the alignment features into the target image;

[0115] Step S104-12: Perform inverse diffusion on the replacement image to generate a target replacement image. The purpose of performing inverse diffusion on the replacement image is to remove Gaussian noise input into the diffusion model, thereby obtaining a clear replacement image.

[0116] The above is a description of an embodiment of an image processing method provided by the present application. In this embodiment, facial images are mainly used as an example for explanation. In fact, the target image and the reference image can be images of the same type. Of course, they can also be images of different types according to different image generation requirement scenarios. In this embodiment, a new facial image is generated by local replacement of facial organs as an example for explanation. In scenes such as buildings and environments, the above method can still be used to replace local image elements to achieve fine-grained image generation, so that image generation no longer depends on the replacement of the overall synchronous movement method. It can not only improve the naturalness of the generated image, but also has the advantage of fine-grained high controllability, so that the layout information from different reference images can be integrated into the target image.

[0117] The above is a detailed description of an embodiment of an image processing method provided by the present application. Corresponding to the above-mentioned embodiment of an image processing method, the present application also discloses an embodiment of an image processing device. Figure 2Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0118] like Figure 2 As shown, Figure 2 : is a structural schematic diagram of an image processing device provided by the present application, the device comprising:

[0119] The first determining unit 201 is used to detect the local position of the target in the target image, determine the target local position information of the target local information in the target image, and shield the target image remaining features of the target local information;

[0120] The second determining unit 202 is configured to input the determined reference local image in the reference image into the first network model for feature extraction to determine the reference local feature according to the detection of the reference local position in the reference image;

[0121] The first generating unit 203 is used to input the target local position information and the reference local feature into the second network model for feature integration, and generate an alignment feature for aligning the reference local feature with the target local position information;

[0122] The second generating unit 204 is used to input the alignment features and the remaining features of the target image into a third network model to generate a replacement image in which the target local features in the target facial image are replaced with reference local features.

[0123] The first determining unit 201 includes: a determining subunit, a first generating subunit, a second generating subunit and an extracting subunit;

[0124] The determination subunit is used to determine the position information of the target local information in the target image according to the detection of the target local position in the target image;

[0125] The first generating subunit is used to generate a target local image including the target local information according to the target local information corresponding to the position information;

[0126] The second generating subunit is used to mask a target local position in the target image according to the target local image to generate a target local masked image;

[0127] The extraction subunit is used to input the target local shielding image into the fourth network model to extract the remaining features of the target image.

[0128] The second determining unit 202 includes: a first determining subunit and a second determining subunit;

[0129] The first determination subunit is used to determine the reference local information corresponding to different reference local positions in the same reference image based on the detection of different reference local positions in the same reference image; or to determine the reference local information corresponding to different reference local positions in different reference images based on the detection of different reference local positions in multiple different reference images respectively;

[0130] The second determining subunit is used to shield the image information except the reference local information in the reference image according to the reference local position, so as to determine a plurality of reference local images.

[0131] The first generating unit 203 includes: a first generating subunit and a second generating subunit;

[0132] The first generating subunit is used to input the reference local features into the second network model to perform latent space feature splicing to generate a local feature splicing image;

[0133] The second generating subunit is used to perform convolution processing on the local feature stitching image, align the local stitching features in the local feature stitching image with the target position information, and generate the alignment features.

[0134] The third generating subunit 204 includes: a determining subunit and a generating subunit;

[0135] The determination subunit is used to input the alignment feature and the target residual feature into the third network model to perform feature addition, and determine a replacement image that fuses the alignment feature into the target image;

[0136] The generating subunit is used to perform inverse diffusion on the replacement image to generate a target replacement image.

[0137] The above is a description of an image processing device provided by the present application. For the specific implementation process of the device, reference may be made to the above method embodiment, which will not be described in detail here.

[0138] Based on the above, the present application also provides a facial image generation method, such as Figure 3 and Figure 4 As shown, the specific implementation process of this method can refer to the embodiment of the above-mentioned image processing method. Figure 3 is a flowchart of a facial image generation method provided by the present application, Figure 4 This is a schematic diagram of the execution process architecture of a facial image generation method provided by this application.

[0139] like Figure 3 and Figure 4 As shown, the facial image generation method includes:

[0140] Step S301: detecting a target facial local position in a target facial image, determining target facial local position information of target facial local information in the target facial image, and shielding remaining features of the target facial image of the target facial local information;

[0141] Step S302: Based on the detection of the reference facial local position in the facial reference image, the reference facial local image in the facial reference image is input into the first network model for feature extraction to determine the reference facial local features;

[0142] Step S303: inputting the target facial local position information and the reference facial local features into a second network model for feature integration, and generating an alignment feature for aligning the reference facial local features with the target facial local position information;

[0143] Step S304: inputting the alignment features and the remaining features of the target facial image into a third network model to generate a replacement image in which the target facial local features in the target facial image are replaced with the reference facial local features.

[0144] Regarding the facial image generation method, reference may be made to the contents of step S101 to step S104 and no further details will be given here.

[0145] It can be understood that the face in the above-mentioned facial image generation method embodiment can be any facial contour information, including AI images.

[0146] like Figure 5 As shown, the present application also provides a structural schematic diagram of a facial image generating device, the device comprising:

[0147] The first determining unit 501 is used to detect the local position of the target face in the target face image, determine the target face local position information of the target face local information in the target face image, and shield the remaining features of the target face image of the target face local information;

[0148] The second determining unit 502 is used to input the determined reference facial local image in the facial reference image into the first network model for feature extraction to determine the reference facial local features according to the detection of the reference facial local position in the facial reference image;

[0149] A first generating unit 503 is used to input the target facial local position information and the reference facial local features into a second network model for feature integration, and generate an alignment feature for aligning the reference facial local features with the target facial local position information;

[0150] The second generating unit 504 is used to input the alignment features and the remaining features of the target facial image into a third network model to generate a replacement image in which the target facial local features in the target facial image are replaced with the reference facial local features.

[0151] The specific implementation process of the facial image generating device can refer to the contents of step S101 to step S104 and will not be described in detail here.

[0152] Based on the above, the present application also provides an image processing system, such as Figure 6 As shown, the Figure 6 60 is a schematic diagram of the structure of an image processing system embodiment provided by the present application. The system embodiment may include: a detection module 601, a first network model 602, a second network model 603, a third network model 604, and a fourth network model 605;

[0153] The detection module 601 is used to perform target local position detection on the input target image, determine the target local position information of the target local information to be replaced in the target image, and the remaining features of the target image after the target local information is masked; and to perform reference local position detection on the input reference image, and determine the reference local image in the reference image;

[0154] The first network model 602 is used to input the reference local image into the first network model to extract features and determine reference local features;

[0155] The second network model 603 is used to input the target local position information and the reference local feature into the second network model for feature integration, and generate an alignment feature for aligning the reference local feature with the target local position information;

[0156] The third network model 604 is used to input the alignment features and the remaining features of the target image into the third network model to generate a replacement image in which the target local features in the target image are replaced with reference local features;

[0157] The fourth network module 605 is used to extract the remaining features of the target image.

[0158] The specific content of the system can be combined with the content of the above steps S101 to S104, which will not be described in detail here.

[0159] Based on the above, the present application also provides a computer storage medium, including a computer program. When the computer program is run on an electronic device, the electronic device executes the relevant steps of the image processing method as described above, and / or executes the relevant steps of the facial image generation method as described above.

[0160] Based on the above, the present application also provides an electronic device, such as Figure 7 As shown, Figure 7 701 is a schematic diagram of the structure of an electronic device provided by the present application. The electronic device embodiment may include: a processor 701 and a memory 702;

[0161] The memory 702 is used to store a program for processing data generated by the electronic device. When the program is read and executed by the processor 701, it executes the relevant steps of the above-mentioned image processing method and / or the relevant steps of the above-mentioned facial image generation method.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0163] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0164] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0165] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0166] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. An image processing method, characterized in that: include: Detecting the local position of the target in the target image, determining the target local position information of the target local information to be replaced in the target image, and shielding the remaining features of the target image of the target local information; According to the detection of the reference local position in the reference image, the reference local image in the reference image is input into the first network model for feature extraction to determine the reference local feature; Inputting the target local position information and the reference local feature into a second network model for feature integration, and generating an alignment feature for aligning the reference local feature with the target local position information; The alignment features and the remaining features of the target image are input into a third network model to generate a replacement image in which the target local features in the target image are replaced with reference local features.

2. The image processing method according to claim 1, characterized in that: The detecting of the local position of the target in the target image, determining the target local position information of the target local information in the target image, and shielding the remaining features of the target image of the target local information include: Determining position information of the local target information in the target image based on detection of the local target position in the target image; generating a target partial image including the target partial information according to the target partial information corresponding to the position information; According to the target partial image, a target partial position in the target image is masked to generate a target partial masked image; The target local masked image is input into the fourth network model to extract the remaining features of the target image.

3. The image processing method according to claim 1, characterized in that: The performing reference local position detection on the reference image to determine the reference local image in the reference image includes: Determine reference local information corresponding to different reference local positions in the same reference image based on detection of different reference local positions in the same reference image; or determine reference local information corresponding to different reference local positions in different reference images based on detection of different reference local positions in multiple different reference images respectively; According to the reference local position, image information other than the reference local information in the reference image is shielded to determine a plurality of reference local images.

4. The image processing method according to claim 3, characterized in that: The step of inputting the reference local image into a first network model to extract features and determine reference local features includes: The multiple reference local images are input into the first network model to extract local features, and the reference local features corresponding to the reference local images are determined.

5. The image processing method according to claim 1, characterized in that: The step of inputting the target local position information and the reference local feature into a second network model for feature integration to generate an alignment feature for aligning the reference local feature with the target local position information includes: Inputting the reference local features into the second network model to perform latent space feature splicing to generate a local feature splicing image; The local feature stitching image is subjected to convolution processing, and the local stitching features in the local feature stitching image are aligned with the target position information to generate the alignment features.

6. The image processing method according to claim 1, characterized in that: The step of inputting the alignment features and the target image remaining features into a third network model to generate a replacement image in which the target local features in the target image are replaced with reference local features includes: Inputting the alignment features and the target remaining features into the third network model to perform feature addition, and determining a replacement image that fuses the alignment features into the target image; The replacement image is subjected to inverse diffusion to generate a target replacement image.

7. An image processing device, characterized in that: include: A first determination unit is used to detect a local position of a target in a target image, determine target local position information of target local information in the target image, and shield remaining features of the target image of the target local information; A second determining unit is used to input the determined reference local image in the reference image into the first network model for feature extraction to determine the reference local feature based on the detection of the reference local position in the reference image; A first generating unit is used to input the target local position information and the reference local feature into a second network model for feature integration, and generate an alignment feature for aligning the reference local feature with the target local position information; The second generating unit is used to input the alignment features and the remaining features of the target image into a third network model to generate a replacement image in which the target local features in the target image are replaced with reference local features.

8. A facial image generation method, characterized in that: include: Detecting a local position of a target face in a target face image, determining target face local position information of the target face local information in the target face image, and shielding remaining features of the target face image of the target face local information; According to the detection of the reference facial local position in the facial reference image, the reference facial local image in the facial reference image is input into the first network model for feature extraction to determine the reference facial local features; Inputting the target facial local position information and the reference facial local features into a second network model for feature integration, and generating an alignment feature for aligning the reference facial local features with the target facial local position information; The alignment features and the remaining features of the target facial image are input into a third network model to generate a replacement image in which the target facial local features in the target facial image are replaced with the reference facial local features.

9. A facial image generating device, characterized in that: include: A first determining unit is used to detect a local position of a target face in a target face image, determine target face local position information of the target face local information in the target face image, and shield remaining features of the target face image of the target face local information; A second determining unit is used to input the determined reference facial local image in the facial reference image into the first network model for feature extraction to determine the reference facial local features based on the detection of the reference facial local position in the facial reference image; A first generating unit is used to input the target facial local position information and the reference facial local features into a second network model for feature integration, and generate an alignment feature for aligning the reference facial local features with the target facial local position information; The second generating unit is used to input the alignment features and the remaining features of the target facial image into a third network model to generate a replacement image in which the target facial local features in the target facial image are replaced with the reference facial local features.

10. An image processing system, characterized in that: include: Detection module, first network model, second network model, third network model, fourth network model; The detection module is used to perform target local position detection on the input target image, determine the target local position information of the target local information to be replaced in the target image, and the remaining features of the target image after the target local information is masked; and to perform reference local position detection on the input reference image, and determine the reference local image in the reference image; The first network model is used to input the reference local image into the first network model to extract features and determine reference local features; The second network model is used to input the target local position information and the reference local feature into the second network model for feature integration, and generate an alignment feature for aligning the reference local feature with the target local position information; The third network model is used to input the alignment features and the remaining features of the target image into the third network model to generate a replacement image in which the target local features in the target image are replaced with reference local features; The fourth network module is used to extract the remaining features of the target image.

11. A computer storage medium, characterized in that: It comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the image processing method according to any one of claims 1 to 6, and / or the facial image generation method according to claim 8.

12. An electronic device, characterized in that: include: processor; A memory for storing a program for processing data generated by an electronic device, wherein when the program is read and executed by the processor, the program executes the image processing method according to any one of claims 1 to 6 and / or the facial image generation method according to claim 8.

Images