Face sketch-photo synthesis method based on multi-condition prior guidance

Abstract

The invention belongs to the technical field of face sketch-photo synthesis, and particularly relates to a multi-condition priori guided face sketch-photo synthesis method, which comprises the following steps of: 1, collecting sketch-photo face image pairs of different people and randomly selected reference photo face images, and preprocessing the images; 2, introducing a semantic feature and texture feature decoupling enhancement network module, and constructing a multi-condition priori guided face sketch-photo synthesis network model; 3, selecting a proper loss function to update network parameters, and training a multi-condition priori guided face sketch-photo synthesis network model by using a sketch face image and a randomly selected reference photo face image; and 4, inputting a to-be-synthesized sketch face image and a randomly selected reference photo face image into the trained face sketch-photo synthesis network model, and outputting a synthesized face photo. According to the method, sketch features of the enhancement conversion process can be decoupled from two angles by using semantic and texture prior features of a reference photo, a face structure can be modeled more completely by using a double helix scanning mode, and color information easy to lose in the fusion process can be supplemented by using color prior; therefore, the synthesized image is more reasonable in facial structure and richer in color.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a multi-condition prior guided sketch-photo synthesis method. BACKGROUND

[0002] The sketch-photo synthesis technology has wide application value in many real application scenarios such as law enforcement investigation and digital entertainment. For example, in the law enforcement field, case handlers usually draw a sketch of a suspect according to the description of a witness, and match it with a photo database to assist in identity recognition. However, due to the significant modal difference between the sketch and the photo, this task belongs to the typical heterogeneous face recognition problem, and it is difficult to achieve consistent alignment and accurate matching of features.

[0003] In the prior art, earlier methods include regression models based on image block matching and sample-based mapping methods. For example, existing researches match image blocks by using a multi-scale Markov random field, or realize the mapping of high-frequency information between sketches and photos by support vector regression. However, such traditional methods have certain limitations in modeling complex structures and details. With the development of deep learning technology, architectures such as convolutional neural networks (CNN) and Transformers have gradually become mainstream. CNN has strong ability in extracting local features, but due to the limitation of the receptive field, it is difficult to capture long-distance dependencies, which affects the restoration of the overall structure. The Transformer architecture has strong global modeling capability, which to some extent makes up for the above defects, but has the problems of large computational overhead and large parameter quantity on high-resolution images, and has the problems of efficiency and scalability.

[0004] Recently, state space models (SSM) have attracted attention again due to their linear time complexity and ability to model long-distance dependencies. Among them, the Mamba architecture introduces a selective scanning mechanism and exhibits good performance in visual tasks such as semantic segmentation and change detection. Existing researches have tried to combine Mamba with CNN and apply it to the sketch-photo synthesis task, and have made certain progress. However, there are still the following problems in the direct application process:

[0005] Current methods often model the semantic and texture features across modalities as a whole, ignoring their independence, resulting in structural artifacts and perceptual inconsistency in the generated images;

[0006] General scanning methods (such as raster scanning, local scanning or linear scanning) have deficiencies in spatial structure modeling, and it is difficult to preserve the texture continuity and hierarchical structure of the image, affecting the output quality;

[0007] The feature fusion mode is too simple, lacks high-level prior guidance (such as semantics, texture or color), causes the loss of key feature information in the fusion process, and affects the fidelity and perception effect of the synthesized image.

[0008] Therefore, there is an urgent need for a new sketch-photo synthesis method with reasonable structure, higher efficiency and more effective fusion mechanism to improve the realism, structure restoration ability and perceptual consistency of images. SUMMARY

[0009] In view of the deficiencies in the prior art, the purpose of the present application is to provide a multi-condition prior guided double helix CNN-Mamba network face sketch-photo synthesis method, which realizes decoupled modeling and fusion optimization of semantic features and texture features in a unified deep model, is conducive to generating target modality images with higher structural fidelity, texture continuity and perceptual quality, and promotes the generation of identity consistent face images. The present application introduces semantic prior, texture prior and color prior to improve the global structure expression and detail restoration ability of the image during synthesis, and at the same time, with the linear time series modeling capability of the state space model Mamba, the cross-modal modeling capability is improved while maintaining efficient calculation. The generated high-quality photo image can be further used to train a heterogeneous face recognition system, thereby improving its recognition accuracy and generalization ability.

[0010] To achieve the above purpose, the present application provides the following technical solutions:

[0011] A multi-condition prior guided face sketch-photo synthesis method, the method comprising the following steps:

[0012] Obtaining a sketch face image to be synthesized and a reference photo face image randomly selected;

[0013] Inputting the sketch face image to be synthesized and the reference photo face image randomly selected into the trained face sketch-photo synthesis network model, and outputting the synthesis result of the synthesized face image;

[0014] The training process of the face sketch-photo synthesis network model comprises:

[0015] Obtaining a sketch-photo pair data set, wherein each pair of samples contains a sketch face image and its corresponding visible light photo image. At the same time, an HSV color extraction network is used to generate three types of auxiliary prior maps from the visible light image, including semantic color map, texture color map and comprehensive color map, which are used as semantic, texture and color prior inputs respectively.

[0016] Multi-scale feature extraction is performed on the input sketch image and the three types of HSV prior maps to extract feature representations at different resolutions for subsequent decoupled enhancement and fusion modeling.

[0017] With sketch images and semantic prior maps as inputs, spatial adaptive normalization semantic modulation and state space model are combined to enhance semantic consistency; with sketch images and texture prior maps as inputs, adaptive instance normalization texture modulation and state space modeling module are combined to improve the ability of detail recovery and edge alignment; multi-scale features of semantic features, texture features and color prior maps are fused, and significant area enhancement and color correction are performed by using compression and excitation module, global average pooling and attention mechanism to generate final high-quality photo image output.

[0018] By optimizing the total loss of the adversarial loss, the cycle consistency loss and the perception loss, the semantic prior feature decoupling enhancement module, the texture prior feature decoupling enhancement module and the feature fusion module of the face sketch-photo synthesis network model are alternately trained until the preset conditions of training are met.

[0019] The sketch face image to be tested is input into the trained multi-condition prior guided face sketch-photo synthesis network model, so as to output the corresponding photo image.

[0020] Compared with the prior art, the method has the following advantages and beneficial effects:

[0021] 1) The structural decoupling of cross-modal features can be realized by double-branch modeling of semantics and texture, and the generated face image has higher structural clarity and detail realism;

[0022] 2) The spatial modeling capability of the face region can be enhanced by combining the double helix scanning path, and compared with the traditional scanning strategy, the key structure information of the face can be better preserved;

[0023] 3) Multi-modal guided fusion of semantics, texture and color can be realized, effectively improving the problems of image color loss and fused information blur;

[0024] 4) The generated high-quality target modal image can be used as a training sample to expand the data distribution of the heterogeneous face recognition model, thereby improving the robustness and cross-modal recognition performance of the recognition model; the model proposed in the application can also be directly used as an image preprocessing step for cross-modal face recognition to improve the recognition performance. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below with reference to the drawings, in which:

[0026] Figure 1 A flowchart of the multi-condition prior guided face sketch-photo synthesis method in one or more embodiments of the application;

[0027] Figure 2A structural schematic diagram of a multi-condition prior guided face sketch-photo synthesis model in one or more embodiments of the present application;

[0028] Figure 3 A structural schematic diagram of a double helix Mamba of the present application;

[0029] Figure 4 A training flow schematic diagram of a multi-condition prior guided face sketch-photo synthesis model in one or more embodiments of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0031] The preferred embodiments of the present application will be described in detail in combination with the drawings.

[0032] Figure 1 A multi-condition prior guided face sketch-photo synthesis method in an embodiment of the present application, as shown in Figure 1 The method comprises the following steps.

[0033] S1, collecting and acquiring sketch-photo face image pairs of different people and randomly selected reference photo face images, and pre-processing the images.

[0034] In the embodiment of the present application, the sketch face image to be synthesized is the face image to be synthesized, and the randomly selected reference photo face image is the face image providing prior information. The sketch face image to be synthesized of different modalities needs to be processed to obtain the final synthesis result of the face image to be synthesized.

[0035] S2, introducing a semantic feature and texture feature decoupling enhancement network module to construct a multi-condition prior guided face sketch-photo synthesis network model.

[0036] S3, selecting a suitable loss function to update the network parameters, and training the multi-condition prior guided face sketch-photo synthesis network model by using the sketch face image and the randomly selected reference photo face image.

[0037] S4, inputting the sketch face image to be synthesized and the randomly selected reference photo face image into the face sketch-photo synthesis network model after training, and outputting a synthesized photo.

[0038] In the embodiment of the present application, considering that the existing face sketch-photo synthesis method only considers the facial features from the overall aspect when synthesizing a face image, the complex structure of the face is not fully processed, and the feature fusion is not sufficient, and the traditional deep convolutional neural network model is difficult to effectively synthesize a better result. Therefore, the modal invariant semantic and texture features are decoupled and enhanced in a unified deep model, and the two tasks promote each other to facilitate the generation of high-quality, identity-keeping target modal face images and supplement the color information easily lost in the decoding process.

[0039] In order to better illustrate the face sketch-photo synthesis network model of the present application, the face sketch-photo synthesis network model and its corresponding training process are described in detail.

[0040] In the embodiment of the present application, as shown in Figure 2 The face sketch-photo synthesis model includes a semantic prior feature decoupling enhancement module, a texture prior feature decoupling enhancement module and a feature fusion module, wherein the semantic prior feature decoupling enhancement module is composed of a double spiral Mamba (DS SSM) and a spatial adaptive normalization residual module (SPADE ResBlock); the texture prior feature decoupling enhancement module is basically the same as the semantic prior feature decoupling enhancement module, and is also composed of a double spiral Mamba (DS SSM) and an adaptive instance normalization residual module (AdaIN ResBlock); the feature fusion module is composed of a plurality of different convolution operations, in addition, an encoder Encoder and a decoder Decoder and a discriminator D are connected before and after the module.

[0041] Figure 3 is a structural diagram of the double spiral Mamba of the present application, as shown in Figure 3 The double spiral Mamba contains three linear layers, a depth separable convolution, two Sigmoid linear unit activation functions and a state space model. The double spiral scanning mechanism we propose is embedded before the state space model to effectively help model the semantic and texture features of the face.

[0042] In the embodiment of the present application, the face sketch-photo synthesis model is trained in the manner of CycleGan, and 70X 70PatchGAN is used as the discriminator.

[0043] Figure 4 The face sketch-photo synthesis network model training flowchart of the embodiment of the present application is shown in Figure 4 The training process of the face sketch-photo synthesis network model includes:

[0044] 101、Obtain the original face image training set under each modality, including the sketch face image set and the photo face image set;

[0045] In the embodiment of the present application, the face images under each modality need to be collected as the training set, including the sketch face image set and the photo face image set and respectively represent the i-th sketch face image and photo face image, and respectively represent the identity label of the i-th sketch face image and photo face image, and may belong to different identities, that is

[0046] For the convenience of description, the face images from the CUHK student dataset can be collected in the embodiment, and the dataset includes the face images of 188 persons, wherein 88 pairs of images are used as the training set, and the rest are used as the test set. The size of all images in the training set is adjusted, and the size of all images is adjusted to 128x128x3.

[0047] 102、Input the original sketch face image and the randomly selected photo face image into the encoder and the color prior feature extractor of the face sketch-photo synthesis network model, use the encoder to extract the feature map of one kind of modality face image, use the color prior feature extractor to extract the semantic, texture and color prior three kinds of feature maps of the other kind of modality face image, and use the extracted features to decouple and enhance the sketch features to help generate the face image after modality conversion;

[0048] In the embodiment of the present application, the input sketch image S is subjected to multi-layer down-sampling convolution processing, and the feature maps on three resolution levels (such as 512x512, 256x256 and 128x128) are extracted, denoted as At the same time, the input HSV semantic prior feature map, texture prior feature map and color prior feature map also extract multi-scale features

[0049] 103、The semantic prior feature decoupling enhancement module takes the sketch image feature and the semantic prior feature as input, first passes through layer normalization, Mamba's double helix state space modeling layer, constructs a bidirectional sequence from the center to the outside and the outside to the center, and models the semantic context information. Subsequently, the spatial adaptive normalization residual module semantic modulation module calculates the scaling and translation parameters guided by the semantics, and realizes the adaptive modulation of the input sketch feature. The module outputs the semantic enhanced feature F SThe texture prior feature decoupling enhancement module and the semantic prior feature decoupling enhancement module are similar in structure, but the texture prior feature The adaptive instance normalization residual module texture modulation mechanism adjusts the mean and variance of the sketch image features, respectively, so that the features are closer to the real photo in texture distribution. The module outputs the texture-enhanced features F T .

[0050] The core components of the two modules, the double helix scanning state space model, have the following structure:

[0051] The input features are first preprocessed by layer normalization, linear activation function, depth separable convolution, and Swish activation function;

[0052] The processed two-dimensional feature map is subjected to central helical scanning and edge reverse scanning to obtain two one-dimensional sequences S1 and S2, respectively;

[0053] The sequence inputs of the two paths are input into the state space model layer for front and back timing modeling;

[0054] The output context-enhanced features are used as the reference for the subsequent modulation module.

[0055] This structure effectively avoids the position information disorder and edge distortion problems caused by traditional scanning methods.

[0056] 104、In the feature fusion stage, the feature fusion module fuses F S and F T by concatenation and convolution, and introduces the multi-scale color features F C of the HSV color prior map for guidance. The feature fusion module includes the following steps:

[0057] The compression and excitation module is introduced to adaptively adjust the channel response weight and emphasize the salient regions;

[0058] The global average pooling module is used to extract global context-aware features;

[0059] The cross-attention mechanism is used to achieve fine registration between semantic-texture features and color features;

[0060] Finally, the fused feature map F O is output, which is used for subsequent image decoding to generate the final visible light face image. The fused feature F O is restored to the original image size by the upsampling decoder module, and the final photo image I O is output. This image maintains high quality in structure restoration, detail clarity, and color consistency, and can be used for training and verification of the face sketch-photo synthesis network model.

[0061] 105. By optimizing the total loss of adversarial loss, cycle consistency loss and perceptual loss, alternately train the semantic prior feature decoupling enhancement module, texture prior feature decoupling enhancement module and feature fusion module of the face sketch-photo synthesis network model until the preset training conditions are met.

[0062] 106. Input the sketched face image to be tested into the trained multi-condition prior-guided face sketch-photo synthesis network model, and then output the corresponding photo image.

[0063] In this embodiment of the invention, another modality of face image I is also required. P Compared with the synthesized modality-transformed face image I O The input images are fed into a discriminator D to determine whether they are synthetic or non-synthetic images, thereby constructing an adversarial loss L. adv ;

[0064] The adversarial loss adopts the adversarial loss function of the original GAN ​​network, and the formula is:

[0065] L adv =E y [(D(y)) 2 ]+E x [(1-D(G(x,r,M(x)))) 2 ];

[0066] Given an image x and its corresponding semantic face mask M(x), image x should be able to be restored to the original image after undergoing a cyclic transformation from domain X to domain Y and back to domain X. Therefore, the cycle consistency loss is defined as follows:

[0067] L cycle =E x [||F(G(x,r y ,M(x)),r x ,M(G(x,r y ,M(x))))-x||1];

[0068] Where r x and r y These represent random reference images from domains X and Y, respectively.

[0069] The perceptual loss is introduced to ensure that the generated face photos are similar to real photos at the semantic feature level:

[0070]

[0071] Among them, C j H j and W j These represent the number of channels, height, and width of the feature map, respectively; φj represents the feature map output extracted by the jth layer of the pre-trained VGG-19 network.

[0072] The total loss function is:

[0073] L = λ adv L adv + λ cycle L cycle + λ per L per

[0074] wherein λ adv , λ cycle , λ per represent the weight values for controlling the importance of the corresponding loss. Through a large number of experimental experiences, λ adv , λ cycle , λ per are respectively set to 1, 0.5, 0.5.

[0075] In the preferred embodiment of the present application, the face sketch-photo synthesis network model can also be tested, and in the testing stage, there are two methods of qualitative testing and quantitative testing to verify the effect of the model.

[0076] In the qualitative testing, the face picture to be converted and any face picture under the to-be-converted mode are input into the face sketch-photo synthesis network model trained, and the face image after the mode conversion can be obtained. The effect of the present application can be verified by directly observing the quality of the synthesized mode conversion face image.

[0077] In the quantitative testing, the generated target mode (visible light) face image is paired with the corresponding real visible light face image, and a pre-trained LPIPS perceptual similarity evaluation model is used for evaluation. The synthesized face image and its corresponding real photo image are input into the pre-trained LPIPS model as input pairs, and the perceptual distance in the deep perceptual space is calculated. The smaller the LPIPS value, the higher the similarity of the generated image and the real image in terms of semantic structure and detail texture, and the closer the generated effect to the real face. By statistically calculating the average LPIPS value on the test set, the performance of the face sketch-photo synthesis model proposed in the present application in terms of image quality can be quantitatively evaluated. Compared with the traditional pixel difference measurement index (such as PSNR, SSIM), LPIPS pays more attention to the consistency of human perception, and can more truly reflect the performance of the synthesized image in terms of structure fidelity and detail restoration.

[0078] The performance of the SSIM, FID, and LPISP index test methods is tested in the experiment. The experimental results of the model on the CUHK student and XM2VTS data sets are shown in Tables 1-2. Ours is the method of the present application.

[0079] Table 1 Experimental results of the model on the CUHK student data set

[0080] Method LPIPS SSIM FID PixPix 0.1873 0.7493 153.4215 CycleGAN 0.2223 0.7007 154.2189 SCAGAN 0.1360 0.8302 78.8664 PS2MAN 0.2176 0.7514 110.7754 DCNP 0.1457 0.7649 66.4384 Ours 0.1083 0.8338 46.3254

[0081] Table 2 Experimental results of the model on the XM2VTS data set

[0082]

[0083]

[0084] The experimental results of the model show that the method proposed in the present application achieves the best in most indicators, indicating that the synthesized image results are better in quality compared to other methods.

[0085] The parameters of the face sketch-photo synthesis model are further optimized through the test stage, so that the face sketch-photo synthesis model synthesizes images more naturally and realistically in the actual inference process, effectively improving the test results of the synthesis results.

[0086] Finally, it should be pointed out that the above preferred embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present application.

[0087] Although embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-condition prior guided sketch-photo synthesis method, characterized in that, The method comprises the following steps: Step 1, obtaining sketch-photo face image pairs of different persons and randomly selected reference photo face images; and constructing a data set after preprocessing the obtained sketch-photo face image pairs; Obtaining feature maps respectively containing semantic, texture and color prior information of the randomly selected reference photo face images, and respectively performing down-sampling on the three types of maps and the sketch face image of the i-th person, to obtain feature maps of different sizes; Step 2, wherein the training process of the face sketch-photo synthesis network model comprises: Obtaining an original face image training set in each mode, including a sketch face image set and a photo face image set; Inputting the original sketch face image and the randomly selected photo face image into the feature decoupling enhancement module of the face sketch-photo synthesis network model, using the semantic prior feature decoupling enhancement module to enhance the semantic features of the face image in the synthesis process using the randomly selected photo face image, using the texture prior feature decoupling enhancement module to enhance the texture features of the face image in the synthesis process using the randomly selected photo face image, and using the enhanced semantic features and texture features and the color prior in the feature fusion module to fuse the features; Inputting the enhanced semantic features and texture features and the color prior features extracted from the randomly selected photo face image into the feature fusion module of the face sketch-photo synthesis network model, fusing the enhanced semantic features and texture features and the color prior features extracted from the randomly selected photo face image, and generating a reconstructed face image using the fused features; Step 3, alternately training the semantic prior feature decoupling enhancement module, the texture prior feature decoupling enhancement module and the feature fusion module of the face sketch-photo synthesis network model by optimizing the total loss sum of the adversarial loss, the cycle consistency loss and the perceptual loss, until the preset condition of training is met; Step 4, inputting the sketch face image to be tested into the trained multi-condition prior guided face sketch-photo synthesis network model, to output a corresponding photo image.

2. The multi-condition prior guided sketch-photo synthesis method of claim 1, wherein, The semantic prior feature decoupling enhancement module and the texture prior feature decoupling enhancement module take into account the complex structure of the face, and respectively decouple and enhance the features from the semantic and texture angles, and the feature fusion module fuses the enhanced semantic features and texture features and the color prior features extracted from the randomly selected photo face image, and supplements the easily missing color information in the synthesis process.

3. The multi-condition prior guided sketch-photo synthesis method of claim 1, wherein, The total loss sum is represented as follows: L = λ adv L adv + λ cycle L cycle + λ per L per wherein L represents a total loss, L adv , L cycle , L per represent an adversarial loss, a cycle consistency loss, a perceptual loss; λ adv , λ cycle , λ per represent a weight value for controlling importance of a corresponding loss.

4. The heterogeneous face recognition method of claim 3, wherein, Each loss is represented as follows: L adv = E y [(D(y)) 2 ] + E x [(1 - D(G(x, r, M(x))) 2 ]; L cycle = E x [||F(G(x, r y , M(x)), r x , M(G(x, r y , M(x)))) - x||1]; Wherein, E represents the expectation of the original face image under each mode, G represents the generator, D represents the discriminator, F represents the inverse generator, x represents the input sample, y represents the corresponding real sample, r represents the randomly selected photo face image, M(x) represents the output of the semantic prior feature decoupling enhancement module and the texture prior feature decoupling enhancement module, ‖‖1 represents the L1 norm, and φ j represents the feature extracted by the feature extraction network at the jth layer, C j , H j , W j represent the channel number, height and width of the jth layer feature map.

Images