Face sketch synthesis method and system based on perception loss

Abstract

The invention provides a face sketch synthesis method and system based on perceptual loss, and relates to the technical field of image processing, the quality of face sketch synthesis is improved by introducing a strategy based on perceptual loss, high and low frequency features of a face image are extracted by adopting a VGG19 network, and weighted fusion of cross-resolution features is realized through an NSRM module. Therefore, the details and overall perception of the facial features are effectively captured and reserved on different scales, the fineness of the image is improved on the local details, the consistency and the natural sense of the image are enhanced on the global features, the distance between the generated image and the real image in the feature space can be better matched by using the perception loss, and the image quality is improved. And comprehensive optimization is carried out in combination with various losses such as adversarial loss and spectral consistency loss, so that the parameters of the generator and the discriminator are adjusted more efficiently in the training process.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for synthesizing face sketches based on perceptual loss. Background Art

[0002] In the field of modern computer vision, face sketch synthesis, as a technology that converts facial images into artistic sketches, has broad application prospects, including law enforcement, art design, and digital entertainment. Traditional face sketch synthesis methods rely on manually designed features and complex image processing algorithms, which are difficult to capture facial details and texture features, resulting in the lack of naturalness and realism in the synthesized sketch images.

[0003] Existing technologies usually use neural network models to improve the quality of sketch synthesis. These models mainly rely on generative adversarial networks (GANs) for end-to-end training, and improve the authenticity of synthesized images through game learning between the generator and the discriminator. These methods have improved the effect of sketch synthesis to a certain extent, and the generated images have good visual effects. However, strategies that rely solely on adversarial loss often ignore high-level perceptual information and have difficulty in effectively fusing image details at different resolutions and feature scales.

[0004] The shortcomings of existing technologies are that they are limited in the feature extraction of facial images and the fineness of generating synthetic sketches. Traditional methods are not effective in processing facial features with varying resolutions or complex details. In addition, using only adversarial loss may result in the generated images being not delicate enough in detail processing and lacking high-level perception of facial features, which ultimately affects the overall quality and authenticity of the sketch image. Therefore, how to effectively integrate multi-scale features and improve the perceived quality has become the main challenge facing current face sketch synthesis technology.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for synthesizing face sketches based on perceptual loss to solve the problems raised in the above-mentioned background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions: A face sketch synthesis method based on perceptual loss, the specific steps include: Step 1: Obtain an existing facial image and a corresponding sketch image of a human face, map the facial image and the corresponding sketch image one by one, and perform facial detection and alignment preprocessing to ensure that the facial image and the sketch image have consistent facial features and the facial features are aligned one by one; Step 2: Use the VGG19 network to extract features from the aligned facial images. Through convolution and nonlinear activation functions layer by layer, the images are converted into feature maps. In the features extracted at each layer, high-pass filters and low-pass filters are used to separate high-frequency detail features and low-frequency global features, and they are processed separately to form high-frequency feature maps and low-frequency feature maps; Step 3: Input the high-frequency and low-frequency feature maps into the NSRM module, and realize the weighted fusion of cross-resolution features through feature expansion and attention weight calculation. Feature expansion copies the original feature map to the batch size so that it can be compared with the current batch samples. The attention mechanism calculates the correlation weight between each feature map to realize the weighted superposition of features and form a unified fusion feature. Step 4: Input the fused features into the generator network. The generator network processes the feature map through the mirrored VGG decoder. The decoder gradually restores the image space structure through multiple layers of inverse convolution, maps the abstract feature vector back to the pixel space, and generates a preliminary synthetic sketch image through layer-by-layer upsampling and convolution operations. Step 5: Input the generated sketch image and the corresponding real sketch image into the discriminator, extract features from the image through the convolution layer, calculate the authenticity score and adversarial loss for each pair of images, and calculate the perceptual loss and spectral consistency loss between the generated image and the real image; Step 6: Based on the loss calculation results, use the back propagation algorithm to update and adjust the parameters of the generator network and the discriminator, repeat the iterative training, set the stop iteration conditions according to the loss calculation results, and obtain the face sketch synthesis model.

[0008] Furthermore, the specific logic of face detection and alignment preprocessing is: The MTCNN network is used to detect the facial image and the corresponding sketch image, and the facial feature points are extracted. The facial feature points include the center of the two eyes and the position of the nose tip. The distance between the eyes and the distance between the eyes and the nose are calculated to obtain the rotation parameter, the scaling parameter and the translation amount. The formula for calculating the distance between the eyes is: in, Represent the corresponding inter-eye distances in the facial image and the sketch image, respectively. Respectively represent the center coordinates of the left and right eye feature points in the facial image, Respectively represent the center coordinates of the left eye and right eye feature points in the sketch image; The formula for calculating the eye-nose distance is: in, They represent the nose tip position and the eye-nose distance between the left and right eyes in the facial image, and they represent the nose tip position and the eye-nose distance between the left and right eyes in the sketch image, Respectively represent the coordinates of the feature points of the nose tip in the facial image and the sketch image; The formula for obtaining the rotation parameters, scaling parameters, and translation is: in, denote the scaling factor and rotation angle respectively, Represent the rotation parameter and scaling parameter respectively, Respectively Direction and The amount of translation in the direction; Based on the affine transformation matrix, all pixels in the original facial image are transformed and the aligned facial image is generated. The formula is: in, They represent the coordinates of the pixel points in the facial image and the coordinates of the pixel points in the aligned facial image respectively.

[0009] Furthermore, the logic of performing face detection and alignment preprocessing also includes adjusting all aligned face images and corresponding sketch images to the same size, which is 512*512 pixels.

[0010] Furthermore, the VGG19 network is used to extract features from the aligned facial images. The images are converted into feature maps through convolution and nonlinear activation functions layer by layer: The VGG19 network used consists of 16 convolutional layers with a 3x3 convolution kernel. The ReLU activation function is applied after each convolution layer to introduce nonlinearity. The 5 maximum pooling layers use a 2x2 pooling window for downsampling. Each pooling layer uses a 2x2 window with a step size of 2, and the maximum pooling layers are respectively after the 2nd, 4th, 7th, 13th and 16th convolutional layers; The first two convolutional layers and the maximum pooling layer behind them capture the facial contour, straight line of the bridge of the nose, outer edge of the lips and edge of the eyes of the aligned facial image, as well as the shadows caused by lighting changes. The second and third convolutional layers and the maximum pooling layer behind them identify the texture details on the skin, as well as the round features of the eyes and nostrils, the thickness and shape of the lips. The fourth to seventh convolutional layers and the maximum pooling layer behind them capture the internal structure of the eyes, including the boundary between the white of the eye and the iris, the curve of the eyelid, and the three-dimensional shape of the bridge of the nose and the opening and closing shape of the mouth. The eighth to twelfth convolutional layers and the maximum pooling layer behind them identify the shape of the face, including the contours of the cheeks and the shape of the chin, and identify the unique facial landmarks, including the location and shape of wrinkles and moles. The twelfth to sixteenth convolutional layers and the maximum pooling layer behind them identify the shape of the eyes and the shape of the chin and cheekbones, and identify and distinguish the facial feature combinations of different individuals, including face shape and facial features. The output feature map of the final maximum pooling layer is the feature map that finally transforms the image.

[0011] Furthermore, when forming high-frequency feature maps and low-frequency feature maps, for each feature map generated by the convolution layer, a 3x3 Laplacian kernel is used as a high-pass filter to extract high-frequency information and obtain a high-frequency feature map. The formula is: in, Respectively represent the VGG19 network The feature map extracted by the convolutional layer and the high-frequency feature map after high-pass filtering, Represents the convolution operation; Use a 3x3 Gaussian kernel as a low-pass filter to extract low-frequency information and obtain a low-frequency feature map. The formula is: in, Indicates the VGG19 network The low-pass filtered low-frequency feature map extracted by the convolutional layer. The high-frequency feature map and low-frequency feature map formed in step 3 are the high-frequency feature map and low-frequency feature map output by the last maximum pooling layer.

[0012] Furthermore, the method for forming a unified fusion feature in step 4 is: High-frequency feature map formed in step 3 And low frequency feature map , use the repeat function to expand it to the batch size , the formula based on is: in, Represent high-frequency feature maps And low frequency feature map After expansion Batch size high-frequency extended feature map and low-frequency extended feature map; The correlation between high-frequency and low-frequency features is calculated using the inner product of the feature map, based on the formula: in, is located in the high-frequency extended feature map High frequency feature map of and the position in the low-frequency extended feature map is Low frequency feature map of The relevance of represents the inner product of the two; For each set of feature vectors , calculate their similarity As a correlation matrix , and: in, represents the total number of all positions in the high-frequency feature map, Represents the total number of all positions in the low-frequency feature map; Use the Softmax function to transform the correlation matrix Normalize each row of to generate a correlation matrix The attention weight of each element in ; Based on the attention weight, the high-frequency and low-frequency features are fused. The attention weight is applied to the high-frequency features and superimposed on the low-frequency features for weighted fusion. The formula is: in, Indicates at location The fused feature vector is Indicates that the position in the high-frequency extended feature map is High frequency feature map of and the position in the low-frequency extended feature map is Low frequency feature map of The attention weights between Representation and location All related locations A collection of; The feature vectors at all positions Combined together to form a complete fusion feature map .

[0013] Furthermore, the specific logic for generating a preliminary synthetic sketch image is as follows: The fused feature map Input the generator network and fusion feature map conduct Convolution operation is used to adjust the number of channels: The decoder uses the mirror structure of VGG and gradually restores the image space through deconvolution and upsampling. In the mirror structure of VGG, the processing of each layer includes deconvolution, ReLU activation and upsampling, specifically: Each layer uses transposed convolution to increase the resolution of the feature map, applies a ReLU activation function to provide non-linear features, and then upsamples the feature map, usually using bilinear interpolation to increase the resolution: Repeat deconvolution, activation and upsampling for each layer of feature maps until the size of the corresponding sketch image is reached; The processed feature map is restored to the image space, and the image generation is completed through transposed convolution and Tanh activation function to obtain a preliminary synthetic sketch image.

[0014] Furthermore, the authenticity score, adversarial loss, perceptual loss, and spectral consistency loss for each pair of images are calculated as follows: The generated synthetic sketch image and the corresponding real sketch image The input is sent to the discriminator network, and the local features of the image are extracted through the continuous convolutional layers in the discriminator network. Each convolutional layer passes through The convolution kernel, batch normalization and LeakyReLU activation function process the input feature map, and the final convolution layer outputs a high-dimensional feature map and ,in To synthesize sketch images The high-dimensional feature map generated by the convolution layer, A high-dimensional feature map generated by a convolutional layer for a real sketch image; The feature map and use The convolution layer is processed, and the Sigmoid function is used to obtain the authenticity scores of the synthetic sketch image and the real sketch image. The formula is: in, denote the authenticity scores of the synthetic sketch image and the real sketch image, respectively. and Represents the feature maps and use The value after the convolution layer is processed. Represents the Sigmoid function; The formulas for calculating adversarial loss and perceptual loss are: in, denote the adversarial loss and perceptual loss between the synthetic sketch image and the real sketch image, respectively. Represent the synthetic sketch images and the corresponding real sketch image Input to the discriminator network, The feature map output by the convolutional layer, represents the Euclidean distance between the two; The formula for calculating spectral consistency loss is: in, represents the spectral consistency loss between the synthetic sketch image and the real sketch image, and represents Fourier transform, which is used to convert the image into the frequency domain, represents the Euclidean distance.

[0015] Furthermore, based on the loss calculation results, the formula for updating and adjusting the parameters of the generator network and the discriminator using the back propagation algorithm is: in, Represent the parameter sets of the generator and discriminator respectively. The parameter sets include the weights of the connections between each layer of neurons in the generator and the discriminator and the bias of the neurons in each layer. denote the learning rates of the generator and the discriminator respectively, Represent the gradients of the generator and the discriminator respectively; The gradients of the generator and discriminator use the chain rule to propagate the gradient back from the output to the input. The calculation formula is: in, represents the total loss, and: in, denote the adversarial loss and perceptual loss between the synthetic sketch image and the real sketch image, respectively. represents the spectral consistency loss between the synthetic sketch image and the real sketch image, denote the adversarial loss weight, the perceptual loss weight, and the consistency loss weight, respectively, and All are not less than 0; When setting the conditions for stopping iteration, set a stopping threshold. If the total loss is less than the stopping threshold, the iterative training will be stopped.

[0016] The present invention further provides a face sketch synthesis system based on perceptual loss, the synthesis system is used to execute the above-mentioned face sketch synthesis method based on perceptual loss, comprising: A data acquisition module is used to obtain a facial image and a corresponding sketch image of an existing human face, wherein the facial image and the corresponding sketch image are mapped one by one, and face detection and alignment preprocessing are performed to ensure that the facial image and the sketch image have consistent facial features and the facial features are aligned one by one; The feature extraction module is used to extract features from the aligned facial images using the VGG19 network. The images are converted into feature maps through convolution and nonlinear activation functions layer by layer. In the features extracted at each layer, high-pass filters and low-pass filters are used to separate high-frequency detail features and low-frequency global features, and they are processed separately to form high-frequency feature maps and low-frequency feature maps. The feature fusion module is used to input the high-frequency and low-frequency feature maps into the NSRM module, and realize the weighted fusion of cross-resolution features through feature expansion and attention weight calculation. The feature expansion copies the original feature map to the batch size so that it can be compared with the current batch samples. The attention mechanism calculates the correlation weight between each feature map to realize the weighted superposition of features and form a unified fusion feature. The image synthesis module is used to input the fused features into the generator network. The generator network processes the feature map through the mirrored VGG decoder. The decoder gradually restores the image space structure through multiple layers of inverse convolution, maps the abstract feature vector back to the pixel space, and generates a preliminary synthetic sketch image through layer-by-layer upsampling and convolution operations; The loss processing module is used to input the generated sketch image and the corresponding real sketch image into the discriminator, extract features from the images through the convolution layer, calculate the authenticity score and adversarial loss of each pair of images, and calculate the perceptual loss and spectral consistency loss between the generated image and the real image; The training analysis module is used to update and adjust the parameters of the generator network and the discriminator based on the loss calculation results using the back propagation algorithm, repeat the iterative training, set the stop iteration conditions according to the loss calculation results, and obtain the face sketch synthesis model.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The present invention improves the quality of face sketch synthesis by introducing a strategy based on perceptual loss, uses the VGG19 network to extract high- and low-frequency features of facial images, and implements weighted fusion of cross-resolution features through the NSRM module, thereby effectively capturing and preserving the details and overall perception of facial features at different scales, which not only improves the fineness of the image in local details, but also enhances the coherence and naturalness of the image in global features. The present invention can better match the distance between the generated image and the real image in the feature space by using perceptual loss. This loss function provides high-level image quality guidance, making the generated sketch image more visually realistic. At the same time, the comprehensive optimization is combined with adversarial loss and spectral consistency loss, making it more efficient to adjust the parameters of the generator and discriminator during the training process. The final generated model can perform well in complex facial image synthesis tasks, solving the shortcomings of the existing technology in detail processing and perceptual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the overall method flow of the present invention; Figure 2 It is a schematic diagram of the overall system structure of the present invention. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0020] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0021] Example

[0022] See also Figure 1 , the present invention provides a technical solution: A face sketch synthesis method based on perceptual loss, the specific steps include: Step 1: Obtain a facial image and a corresponding sketch image of an existing human face, map the facial image and the corresponding sketch image one by one, and perform facial detection and alignment preprocessing to ensure that the facial image and the sketch image have consistent facial features and the facial features are aligned one by one.

[0023] In this embodiment, the specific logic of performing face detection and alignment preprocessing is: The MTCNN network is used to detect facial images and corresponding sketch images, and to extract facial feature points. MTCNN can simultaneously perform facial detection and locate facial feature points, and is suitable for tasks that require high-precision feature points. The facial feature points include the center of the two eyes and the position of the nose tip. The eyes and nose are the most stable and easily recognizable parts of facial features, and are less affected by changes in facial expressions during alignment. The distance between the eyes and the distance between the eyes and nose are calculated to obtain rotation parameters, scaling parameters, and translation. The formula for calculating the distance between the eyes is: in, Represent the corresponding inter-eye distances in the facial image and the sketch image, respectively. Respectively represent the center coordinates of the left and right eye feature points in the facial image, Respectively represent the center coordinates of the left eye and right eye feature points in the sketch image.

[0024] The formula for calculating the eye-nose distance is: in, Respectively represent the nose tip position and the eye-nose distance between the left eye and the right eye in the facial image, They represent the nose tip position and the eye-nose distance between the left eye and the right eye in the sketch image, respectively. Respectively represent the coordinates of the feature points of the nose tip in the facial image and the sketch image. The distances between the eyes and the nose in the facial image and the sketch image are calculated using the Euclidean distance formula. These distances are used to calculate the scaling ratio and rotation angle so that the two images can be geometrically transformed in a consistent manner.

[0025] The formula for obtaining the rotation parameters, scaling parameters, and translation is: in, denote the scaling factor and rotation angle respectively, Represent the rotation parameter and scaling parameter respectively, Respectively Direction and The amount of translation in the direction, the scaling factor is calculated by the ratio of the distance between the eyes and the distance between the eyes and the nose in the two images, which ensures that the image resizing remains relatively consistent, and the rotation angle is obtained by calculating the difference in the tilt angle of the eye line. This angle adjustment makes the eyes of the two images horizontally aligned.

[0026] Based on the affine transformation matrix, all pixels in the original facial image are transformed and the aligned facial image is generated. The formula is: in, They represent the coordinates of the pixels in the facial image and the aligned facial image respectively. The affine transformation matrix is ​​a 2D affine transformation that combines rotation, scaling, and translation. Through the affine transformation, the facial and sketch images can be accurately aligned at each key point.

[0027] Preprocessing and aligning the facial image and its corresponding sketch image is a crucial step, which mainly involves accurately detecting and extracting facial feature points to ensure that the two forms of images can maintain consistency in spatial features. This embodiment uses the MTCNN network to perform facial detection and feature point extraction because MTCNN can effectively identify key feature points of the face, such as the positions of the eyes and nose, which play an important role in the image alignment process.

[0028] The extracted feature points are used to calculate the geometric relationship between the facial image and the sketch image, including the distance between the eyes and the distance between the eyes and the nose. These distances are calculated using the Euclidean formula and used to evaluate the relative size and rotation angle of the two images. From these distances, the required scaling factor and rotation angle can be determined to make the necessary adjustments during the alignment process. This geometric calculation ensures that the facial structure of the two images maintains the same proportions and orientation during the transformation.

[0029] The parameters of the affine transformation are determined based on the calculated rotation angle and scaling factor. Specifically, the rotation and scaling parameters are calculated, as well as the required translation amount, in order to perfectly align the feature points of the two images. The rotation parameters are calculated by the difference in the tilt angles of the eyes, while the scaling parameters are determined by the ratio of the distance between the eyes and the distance between the eyes and the nose. The translation amount is obtained by adjusting the position of the starting feature point. These steps take into account the size, orientation, and position changes of the images.

[0030] By constructing an affine transformation matrix, these parameters are applied to the original facial image, and pixel-level coordinate transformation is performed to generate an aligned facial image. This transformation implements comprehensive adjustments of rotation, scaling, and translation to ensure accurate alignment of the two images at key feature points. This processing step not only improves the effect of subsequent model training, enabling the generated model to better capture and transform facial features, but also ensures the stability and consistency of the generated results.

[0031] Furthermore, the logic for performing face detection and alignment preprocessing also includes adjusting all aligned face images and corresponding sketch images to the same size, which is 512*512 pixels. When all images are adjusted to a uniform 512x512 pixel size, image scaling techniques such as bilinear interpolation or bicubic interpolation are used to adjust the aligned face images and sketch images to the desired size. This adjustment method can maintain image quality while ensuring that each image can still clearly display its key features after scaling. If the image needs to change its aspect ratio during scaling to the target size, it can be selected to fill the edges of the image or crop from the center to adapt to the new size to try to maintain the integrity of the facial features.

[0032] Batch processing all images to ensure that all images go through the same preprocessing steps, which can provide consistent input conditions for subsequent model training and improve the learning effect and generation quality of the model. This standardization process helps reduce the deviation that may be caused by differences in input size during the learning process of the model, thereby improving the stability and consistency of the generated results.

[0033] Step 2: Use the VGG19 network to extract features from the aligned facial images. Through convolution and nonlinear activation functions layer by layer, the images are converted into feature maps. In the features extracted at each layer, high-pass filters and low-pass filters are applied to separate high-frequency detail features and low-frequency global features, and they are processed separately to form high-frequency feature maps and low-frequency feature maps.

[0034] Use the VGG19 network to extract features from the aligned facial images, and convert the images into feature maps layer by layer through convolution and nonlinear activation functions: The VGG19 network used consists of 16 convolutional layers with a 3x3 convolution kernel. The ReLU activation function is applied after each convolution layer to introduce nonlinearity. The 5 maximum pooling layers use a 2x2 pooling window for downsampling. Each pooling layer uses a 2x2 window with a step size of 2, and the maximum pooling layers are respectively after the 2nd, 4th, 7th, 13th and 16th convolutional layers; The first two convolutional layers and the maximum pooling layer following them capture the facial contour, the straight line of the nose bridge, the outer edge of the lips and the edge of the eyes of the aligned facial image, as well as the shadows caused by lighting changes. The second and third convolutional layers and the maximum pooling layers following them identify the texture details on the skin, as well as the roundness of the eyes, the roundness of the nostrils, and the thickness and shape of the lips. The fourth to seventh convolutional layers and the maximum pooling layers following them capture the internal structure of the eyes, including the boundary between the whites of the eyes and the irises, the curves of the eyelids, and the three-dimensional shape of the nose bridge and the opening and closing shape of the mouth. The eighth to twelfth convolutional layers and the maximum pooling layers following them recognize the shape of the face, including the contours of the cheeks and the shape of the chin, and recognize unique facial landmarks, including the location and shape of wrinkles and moles. The twelfth to sixteenth convolutional layers and the maximum pooling layers following them recognize the shape of the eyes and the shape of the chin and cheekbones, and recognize and distinguish the combination of facial features of different individuals, including face shape and facial features.

[0035] The output feature map of the last maximum pooling layer is the feature map that will eventually transform the image. VGG19 consists of 16 convolutional layers. Each convolutional layer uses a 3x3 convolution kernel to ensure that local features are captured during the convolution process. The ReLU activation function is used to introduce nonlinearity to help the network learn more complex patterns. The input image passes through the convolutional layer to generate a feature map, which is then transformed nonlinearly through the ReLU activation function. In this way, the information extracted by each convolutional layer is processed nonlinearly before entering the next layer.

[0036] The max pooling layer uses a 2x2 window and a stride of 2 to reduce the size of the feature map while retaining the most salient features. Pooling not only reduces the dimension of the feature map, but also enhances invariance. Max pooling is applied after the 2nd, 4th, 7th, 13th, and 16th convolutional layers. These pooling layers help to gradually abstract higher-level features, such as from simple edges and textures to more complex shapes and facial features.

[0037] Each layer of VGG19 gradually captures different feature details. The first few layers focus on capturing local edges and simple textures, the middle layers focus on medium-scale features such as facial contours and shapes, and the last few layers focus on high-level abstract features such as overall facial structure and unique features. In the process from shallow to deep layers, the network extracts low-level to high-level features. High-pass filters and low-pass filters are applied to extract high-frequency details (such as textures and edges) and low-frequency global features (such as general shape and facial layout), respectively, and then these features are processed separately to form high-frequency feature maps and low-frequency feature maps. The feature map output by the last layer of pooling is a high-level abstract representation of the entire image, integrating features at all levels.

[0038] The combination of convolution and ReLU enables the network to effectively handle nonlinear problems and capture complex features. The use of small 3x3 convolution kernels helps to learn image features in a more fine-grained manner. By increasing the depth of the network layer by layer, VGG19 can effectively decompose and learn different hierarchical structures in the image, so that feature aggregation can be performed at different scales. Separating high-frequency and low-frequency features enables the system to process details and global structures separately, providing richer feature representations. High-frequency features capture subtle details, while low-frequency features maintain overall structural consistency.

[0039] VGG19 achieves detailed feature extraction through deep network structure and layer-by-layer pooling. Shallower or simpler networks can capture more details and semantic information, providing stronger robustness and accuracy for facial recognition and generation tasks. In this way, the network performs well in processing complex facial features, especially in dealing with various postures, lighting and expression changes.

[0040] Furthermore, when forming high-frequency feature maps and low-frequency feature maps, for each feature map generated by the convolution layer, a 3x3 Laplacian kernel is used as a high-pass filter to extract high-frequency information and obtain a high-frequency feature map. The formula is: in, Respectively represent the VGG19 network The feature map extracted by the convolutional layer and the high-frequency feature map after high-pass filtering, Represents the convolution operation; Use a 3x3 Gaussian kernel as a low-pass filter to extract low-frequency information and obtain a low-frequency feature map. The formula is: in, Indicates the VGG19 network The low-pass filtered low-frequency feature map extracted by the convolutional layer.

[0041] The high-frequency feature map and low-frequency feature map formed in step 3 are the high-frequency feature map and low-frequency feature map output by the last maximum pooling layer. By separating the high-frequency and low-frequency information of the image through high-pass and low-pass filters, a richer and more targeted feature representation can be obtained. The high-frequency features are related to the edge and detail information in the image. The use of the Laplacian kernel can effectively extract this high-frequency information. This process enhances the local details and edge information of the image. The Laplacian kernel highlights the edges and details by differentially processing the surrounding pixels.

[0042] The low-frequency features capture information such as the overall structure and color distribution of the image. The Gaussian kernel is a common smoothing filter that is used to remove high-frequency noise while retaining global information. The Gaussian kernel smoothes the image and emphasizes the overall shape and distribution. By performing the above high-pass and low-pass filtering operations on each convolutional layer, high-frequency and low-frequency feature maps are obtained respectively. Finally, these operations are performed on the feature map output by the last maximum pooling layer to form the final high-frequency feature map and low-frequency feature map.

[0043] Using the Laplacian kernel to extract edge and detail features can help the model focus on local changes in the image, such as facial contours and detailed textures. Smoothing the feature map with the Gaussian kernel and removing high-frequency noise can help the model focus on the overall structure and shape of the image. The separation and combination of high-frequency and low-frequency features provides a multi-scale feature representation method. Compared with the traditional method that only relies on a single feature representation, it can capture the image information more comprehensively and improve the robustness and accuracy of the model.

[0044] Step 3: Input the high-frequency and low-frequency feature maps into the NSRM module, and realize the weighted fusion of cross-resolution features through feature expansion and attention weight calculation. Feature expansion copies the original feature map to the batch size so that it can be compared with the current batch samples. The attention mechanism calculates the correlation weights between each feature map to realize the weighted superposition of features and form a unified fusion feature.

[0045] In this embodiment, the method for forming a unified fusion feature in step 4 is: High-frequency feature map formed in step 3 And low frequency feature map , use the repeat function to expand it to the batch size , the formula based on is: in, Represent high-frequency feature maps And low frequency feature map After expansion Batch size of high-frequency expanded feature maps and low-frequency expanded feature maps.

[0046] Assuming high frequency feature map And low frequency feature map The shape is ,in is the number of channels, are the height and width of the feature map, respectively, expanded on the batch dimension. and There is usually no batch dimension before the repetition operation, so it needs to be expanded to , so that it can be expanded along the first dimension (the batch dimension).

[0047] After the repeat function operation, All have shape , that is, the batch size is This allows each sample in the batch to have the same set of high-frequency and low-frequency feature maps. By expanding the feature map to the batch size, it ensures that the feature map of each sample can be consistently processed in subsequent correlation calculations.

[0048] The correlation between high-frequency and low-frequency features is calculated using the inner product of the feature map, based on the formula: in, is located in the high-frequency extended feature map High frequency feature map of and the position in the low-frequency extended feature map is Low frequency feature map of The relevance of represents the inner product of the two.

[0049] In this process, we calculate the correlation between high-frequency and low-frequency feature maps to capture the mutual information between them. Specifically, the correlation between features is measured by the inner product of the feature maps. The following is a detailed explanation of this process: The inner product is used here to calculate the similarity between two feature maps, i.e., feature vectors. A higher inner product value means that the two feature vectors are more similar in direction, that is, the information they contain is complementary in some sense.

[0050] Location Refers to a specific position in the high-frequency extended feature map. This position can be understood as a specific spatial coordinate of the feature map. The position refers to A specific position in the low-frequency extended feature map corresponds to the same spatial coordinate system, and the corresponding position of the high-frequency and low-frequency feature maps is determined. , this is each spatial position of the feature map. For each pair of positions, the vectors of the high-frequency feature map and the low-frequency feature map are extracted, and their inner product is calculated. Each pixel channel of the feature map is element-by-element multiplied and summed. The calculated inner product values ​​form a correlation matrix. Each element of this matrix corresponds to the similarity between a pair of high-frequency and low-frequency feature map positions. The inner product provides a simple and effective way to evaluate the similarity between two sets of features. Through this calculation, the complementary relationship between the detail information in the high-frequency features and the global information in the low-frequency features can be captured, and the multi-scale features of the image can be effectively identified and utilized, which improves the robustness and accuracy of the model in feature representation. Compared with simple feature concatenation or superposition, the use of inner products can better capture the relationship between different features and improve the performance of fused features in subsequent steps, especially when the attention mechanism is applied.

[0051] The feature map can be regarded as a multi-channel two-dimensional matrix. In a convolutional neural network, each feature map usually has multiple feature channels. In the representation of a specific position in the high-frequency extended feature map, this position can be represented by a two-dimensional coordinate Indicates that are the row and column indices of the feature map, for each pair of specific positions , extract the eigenvectors at these positions and perform inner product calculations. Here, and Refers to the cross-correlation between feature vectors at the same or corresponding positions in the high-frequency and low-frequency feature maps.

[0052] For each set of feature vectors , calculate their similarity As a correlation matrix , and: in, represents the total number of all positions in the high-frequency feature map, Represents the total number of all positions in the low-frequency feature map, the matrix is a real matrix whose size is OK Column, here and It represents the number of spatial positions of high-frequency and low-frequency feature maps, which is related to the spatial dimension of the feature map. In a convolutional neural network, a feature map has three dimensions: number of channels, height, and width. In this case, the number of spatial positions of the feature map is the product of height and width. Therefore, the correlation matrix Each element in Indicates the high-frequency feature map The position and the low-frequency feature map The similarity between the locations.

[0053] Use the Softmax function to transform the correlation matrix Normalize each row of to generate a correlation matrix The purpose of applying the Softmax function to normalize each row of the correlation matrix is ​​to convert each element in the row into a probability value between 0 and 1, and the sum of the probability values ​​of the entire row is 1. This normalization allows the features to be weighted in the attention mechanism to highlight important features.

[0054] The Softmax function converts the elements of the correlation matrix into probability values ​​between 0 and 1. The relative importance of each position in the entire row is highlighted. Normalizing each row ensures that the comparisons between different rows are independent, which is suitable for use in the attention mechanism. The sum of the similarities between each position of the high-frequency feature map and all positions of the low-frequency feature map is 1.

[0055] Based on the attention weight, the high-frequency and low-frequency features are fused. The attention weight is applied to the high-frequency features and superimposed on the low-frequency features for weighted fusion. The formula is: in, Indicates at location The fused feature vector is Indicates that the position in the high-frequency extended feature map is High frequency feature map of and the position in the low-frequency extended feature map is Low frequency feature map of The attention weights between Representation and location All related locations A collection of .

[0056] Indicates at location The fused feature vector of . Specifically, Refers to a specific position or index in a feature map. A feature map is usually represented as a three-dimensional tensor in a convolutional neural network. The number of channels is the depth of the feature map, and the height and width are the spatial dimensions of the feature map. In a feature map, the spatial position is usually represented by a two-dimensional index, which is a combination of height and width. In order to simplify representation and calculation, the two-dimensional spatial position of the feature map is flattened into a one-dimensional index. , Indicates a specific location This means that for each position of the feature map, a corresponding fusion feature vector will be calculated, taking into account the contribution of high-frequency and low-frequency features. is a location set, indicating the location All related locations A collection of positions in which the positions affect or participate in the positions Calculation of features.

[0057] The core idea of ​​the attention mechanism is to assign different weights to different positions in the feature map, thereby emphasizing important information and reducing the influence of unimportant information. It is necessary to calculate the attention weight of each position pair, which is based on the correlation between high-frequency and low-frequency feature maps. A normalization process is used to ensure that all attention weights are equal for each position. The sum is 1, and for each low-frequency feature map position, traverse the set For each position in , and add the results together, superimpose the weighted high-frequency information with the low-frequency features to obtain the fused feature vector. is the correlation matrix The elements in .

[0058] The introduction of the attention mechanism allows the model to dynamically assign weights according to the relevance of features, making the model more flexible in capturing important features. By weighting high-frequency information into low-frequency information, the model can simultaneously utilize local details and global context in the same feature representation. This fusion method can usually generate more expressive feature representations and improve the generalization ability of the model.

[0059] The feature vectors at all positions Combined together to form a complete fusion feature map ,When combining, each feature vector is arranged according to its corresponding spatial position, and a new fused feature map is constructed, ensuring that each feature vector retains its original spatial position, so that the fused feature map can be used in subsequent convolution operations without losing spatial information.,By retaining their respective positions, the combined feature map can better perform subsequent feature processing and analysis, such as convolution, pooling, or further application of attention mechanism.

[0060] Step 4: Input the fused features into the generator network. The generator network processes the feature map by mirroring the VGG decoder. The decoder gradually restores the image space structure through multiple layers of inverse convolution, maps the abstract feature vector back to the pixel space, and generates a preliminary synthetic sketch image through layer-by-layer upsampling and convolution operations.

[0061] In this embodiment, the specific logic of generating a preliminary synthetic sketch image is as follows: The fused feature map Input the generator network and fusion feature map conduct Convolution operation is used to adjust the number of channels: The decoder uses the mirror structure of VGG and gradually restores the image space through deconvolution and upsampling. In the mirror structure of VGG, the processing of each layer includes deconvolution, ReLU activation and upsampling, specifically: Each layer uses transposed convolution to increase the resolution of the feature map, applies a ReLU activation function to provide non-linear features, and then upsamples the feature map, usually using bilinear interpolation to increase the resolution: Repeat deconvolution, activation and upsampling for each layer of feature maps until the size of the corresponding sketch image is reached; The processed feature map is restored to the image space, and the image generation is completed through transposed convolution and Tanh activation function to obtain a preliminary synthetic sketch image.

[0062] In this embodiment, the high-dimensional feature map is converted into a preliminary synthetic sketch image through the mirror VGG decoder in the generator network. This process uses inverse convolution and upsampling techniques to gradually restore the spatial structure of the image, and introduces complex feature representation through nonlinear activation functions. First, The convolution operation adjusts the number of channels of the feature map to match the input requirements of the decoder. Then, the decoder processes the feature map layer by layer, improving the resolution through inverse convolution, adding nonlinear features through ReLU activation, and finally gradually restoring the spatial details of the image through upsampling technology.

[0063] The fused feature map Input to the decoder part of the generator network and perform The purpose of this step is to adjust the number of channels of the feature map to match the needs of the decoder. Convolution is a lightweight operation that does not change the spatial size of the feature map but only adjusts the number of channels.

[0064] The decoder uses the mirror structure of the VGG network, performing inverse convolution, ReLU activation and upsampling operations layer by layer. Inverse convolution is used to increase the spatial resolution of the feature map, and the ReLU activation function, as a nonlinear transformation, helps the model capture richer features. Upsampling further improves the resolution of the image through techniques such as bilinear interpolation.

[0065] In each layer, after the feature map is deconvolved, it will have a higher spatial resolution, but some details may be lost. The ReLU activation function is used to retain positive features and filter out negative values, which helps to maintain the sparsity and diversity of features. Subsequently, through upsampling, the size of the feature map is further increased, gradually approaching the final target image size.

[0066] The processed high-dimensional feature map is restored to the pixel space through transposed convolution, and the value range is adjusted to [-1,1] using the Tanh activation function, which is consistent with the pixel value range of the real image. This is done to ensure that the output synthetic sketch image has appropriate contrast and brightness characteristics.

[0067] pass Convolution not only effectively adjusts the number of channels of the feature map, but also retains spatial information, ensuring the flexibility and adaptability of subsequent processing. The layer-by-layer decoding strategy can effectively restore the spatial structure of the image and ensure the gradual improvement of resolution and details. The use of transposed convolution and upsampling allows the decoder to reconstruct high-quality images from the features. The combination of ReLU and Tanh activation functions not only maintains the complexity of the features, but also ensures the stability of the pixel values ​​of the output image. ReLU can prevent the gradient from disappearing, while Tanh provides an output within a range, making the result more natural. Generating sketch images from high-dimensional features can maintain the spatial and detail information of the original image to a large extent, improving the visual effect and realism of the generated sketch. Compared with other technologies, this method has advantages in detail preservation and global structure reconstruction.

[0068] Step 5: Input the generated sketch image and the corresponding real sketch image into the discriminator, extract features from the image through the convolutional layer, calculate the authenticity score and adversarial loss for each pair of images, and calculate the perceptual loss and spectral consistency loss between the generated image and the real image.

[0069] In this embodiment, the method for calculating the authenticity score, adversarial loss, perceptual loss and spectral consistency loss of each pair of images is as follows: The generated synthetic sketch image and the corresponding real sketch image The input is sent to the discriminator network, and the local features of the image are extracted through the continuous convolutional layers in the discriminator network. Each convolutional layer passes through The convolution kernel, batch normalization and LeakyReLU activation function process the input feature map, and the final convolution layer outputs a high-dimensional feature map and ,in To synthesize sketch images The high-dimensional feature map generated by the convolution layer, A high-dimensional feature map generated by a convolutional layer for a real sketch image; The feature map and use The convolution layer is processed, and the Sigmoid function is used to obtain the authenticity scores of the synthetic sketch image and the real sketch image. The formula is: in, denote the authenticity scores of the synthetic sketch image and the real sketch image, respectively. and Represents the feature maps and use The value after the convolution layer is processed. Represents the Sigmoid function; The formulas for calculating adversarial loss and perceptual loss are: in, denote the adversarial loss and perceptual loss between the synthetic sketch image and the real sketch image, respectively. Represent the synthetic sketch images and the corresponding real sketch image Input to the discriminator network, The feature map output by the convolutional layer, represents the Euclidean distance between the two; The formula for calculating spectral consistency loss is: in, represents the spectral consistency loss between the synthetic sketch image and the real sketch image, and represents Fourier transform, which is used to convert the image into the frequency domain, represents the Euclidean distance.

[0070] An adversarial network framework is used to evaluate the quality of the generated sketch images and the real sketch images. The task of the discriminator network is to distinguish the synthetic images from the real images. By processing the image pairs and calculating different types of losses, the synthetic images are ensured to be as close to the real images as possible visually. The generated sketch images and the real sketch images are input to the discriminator through a series of The convolutional layer extracts features. Each convolutional layer is followed by batch normalization and LeakyReLU activation to maintain the stability of training and the nonlinearity of features. The final high-dimensional feature maps represent the feature encoding of synthetic and real images respectively. These feature maps are obtained by The convolutional layer and Sigmoid activation function are used to generate the authenticity score.

[0071] The adversarial loss is to enhance the competition between the generator and the discriminator, encouraging the generator to generate realistic sketch images. It is a classic adversarial loss. The former encourages the discriminator to make high-realism judgments on real images, and the latter prompts the generator to improve the realism of synthetic images. The perceptual loss evaluates the structural similarity through feature comparison of the convolutional layer and calculates the Euclidean distance between feature maps to ensure that the generated image is perceptually consistent with the real image, making up for the shortcomings of pixel-level metrics. The spectral consistency loss compares the frequency domain information through Fourier transform to ensure that the frequency components of the synthetic image are consistent with the real image, thereby improving the details and overall visual quality of the image.

[0072] The combination of convolutional layers and Sigmoid functions enables the discriminator to effectively extract and evaluate complex features and improve the discrimination ability. Convolution compresses high-dimensional features into a single authenticity score, simplifying the distinction between synthetic and real images. Adversarial loss is a key part in the adversarial network framework, encouraging the generator to improve the quality of synthetic images and approach the real data distribution. This setting promotes mutual improvement between the generator and the discriminator, improving the authenticity of the generated images. The use of perceptual loss can make up for the shortcomings of pixel-level loss, especially in image generation, where only pixel differences may not capture high-level semantic information. Perceptual loss ensures that the structure and details of the image are consistent with the real image by comparing the feature space.

[0073] Frequency domain analysis provides a method to measure image similarity from another dimension. Spectral consistency loss helps the generator retain key texture and edge information and enhance the image detail retention ability by ensuring the consistency of synthetic and real images in the frequency domain. Through the design of the loss function, the generator can be effectively trained to generate synthetic sketch images that are highly consistent with real images in both visual and structural aspects.

[0074] Step 6: Based on the loss calculation results, use the back propagation algorithm to update and adjust the parameters of the generator network and the discriminator, repeat the iterative training, set the stop iteration conditions according to the loss calculation results, and obtain the face sketch synthesis model.

[0075] In this embodiment, based on the loss calculation result, the formula for updating and adjusting the parameters of the generator network and the discriminator using the back propagation algorithm is: in, Represent the parameter sets of the generator and discriminator respectively. The parameter sets include the weights of the connections between each layer of neurons in the generator and the discriminator and the bias of the neurons in each layer. Represent the learning rates of the generator and the discriminator, respectively, and determine the step size of each update. Denote the gradients of the generator and discriminator, respectively. These gradients represent the sensitivity of the loss function to changes in the parameters of the generator and discriminator under the current parameter settings.

[0076] Represents the set of all trainable parameters inside the generator and discriminator respectively. In a neural network, each layer of neurons is connected to the previous layer through certain weights. These weights determine the strength of the input received by each neuron. During the training process, these weights are constantly adjusted to optimize the output of the network. The bias is an additional parameter that is adjusted together with the weights. It can be regarded as the output value of the neuron when there is no input, which is used to adjust the output of the model to better fit the data.

[0077] The gradients of the generator and discriminator use the chain rule to propagate the gradient back from the output to the input. The calculation formula is: in, represents the total loss, and: in, denote the adversarial loss and perceptual loss between the synthetic sketch image and the real sketch image, respectively. represents the spectral consistency loss between the synthetic sketch image and the real sketch image, denote the adversarial loss weight, the perceptual loss weight, and the consistency loss weight, respectively, and Are not less than 0.

[0078] Using gradient descent to update parameters is a standard practice for deep learning optimization. Through accurate gradient calculation and appropriate learning rate, the model can effectively converge to an optimal solution. Adjusting the learning rate is crucial to convergence speed and stability, helping to avoid too large or too small update steps. Calculating the gradient through the chain rule ensures that each part of the loss function is correctly propagated to the corresponding parameters. This method enables the model to more accurately identify the direction of parameter updates and improve training results.

[0079] Combining multiple loss functions for training generative adversarial networks is to improve the quality of generated images in multiple aspects. Adversarial loss is used to improve the authenticity of the generator, perceptual loss ensures the semantic consistency of the synthesized image, and spectral consistency loss ensures the preservation of frequency domain characteristics. The weights between the losses provide a flexible mechanism that can be adjusted for specific task requirements. By integrating multiple loss types, the model is not only trained on the basis of discriminating between generation and reality, but also ensures the quality of the output image at multiple levels. Compared with the method of using only a single loss function, this combined strategy can capture more feature details and improve the visual consistency and realism of the generated image.

[0080] Use random initialization to set the weights in the network. The weights can be randomly drawn from a standard normal distribution or randomly sampled from a uniform distribution. The bias is initialized to 0 because the role of the bias is to adjust the output rather than breaking the symmetry by randomization. At the beginning Choose a small constant value, such as 0.001, 0.0001, etc. This value can be selected through experiments to ensure fast convergence without causing excessive fluctuations. Use adaptive optimization algorithms such as Adam, RMSprop, etc. to automatically adjust the learning rate during training to better adapt to different training stages. If the loss does not decrease or decreases very slowly during training, you can try to reduce the learning rate. If the training process fluctuates greatly and the model is unstable, you may need to reduce the learning rate.

[0081] Start with equal weights when determining, and then adjust based on experimental results. Adjust each weight systematically to observe the impact on the quality of generated images and training stability. If realism and the game between the generator and the discriminator are the main concerns, you can increase it appropriately. , if structural consistency and preservation of semantic information are key, such as in facial feature details, , if frequency and detail feature preservation (such as edges and textures) are important, then you need to focus on If the weight of a loss is too large, the model may focus too much on optimizing the loss and ignore other important features. During the training process, monitor the changing trends of various loss items and adjust the weights to balance their decline rate and final impact. You can choose to adjust the weights within a range. Try combinations and select the best combination through training and performance.

[0082] When setting the conditions for stopping iterations, set a stop threshold. If the total loss is less than the stop threshold, stop iterative training and monitor the total loss. In each training iteration, the current total loss is calculated and recorded to see whether it shows a trend of continuous decline or stabilization. To this end, a stopping threshold can be set as an indicator. When the loss is lower than this value, the training can be terminated. This threshold is usually set based on experience or experiments to ensure that the model achieves satisfactory performance within a reasonable time.

[0083] There are several ways to set the stopping threshold. You can set an empirical threshold based on the performance on the validation set to ensure good performance on the validation set. You can also set a relative change threshold, that is, when the relative change in loss is less than a certain ratio, it is considered close to convergence. In addition, set an absolute loss value and stop training when the loss drops below this value. At the same time, combined with the early stopping method, you can stop training when the verification loss no longer improves in several iterations to avoid overfitting.

[0084] Reasonable stop threshold settings can improve the efficiency and effect of model training, balance training time and performance, and make the model more generalizable and stable. Through dynamic adjustment and experimental verification, it is ensured that training is stopped in time when the model performance reaches the optimal state. This not only saves computing resources, but also improves the reliability and performance of the model in practical applications.

[0085] The process of obtaining a face sketch synthesis model is to train a deep neural network to convert an input face image into a high-quality sketch image. The training process involves using a generator-discriminator architecture, where the generator is responsible for generating sketch images, while the discriminator evaluates the difference between the generated sketch image and the real image. This training process continuously optimizes the generator so that its output is closer to the real sketch, and finally forms a stable and effective model.

[0086] In this process, perceptual loss plays a key role. Perceptual loss measures the difference between generated images and real images through features extracted by the pre-trained network. This loss focuses not only on pixel-level accuracy, but more importantly on the overall perceptual quality and detail features of the image. This allows the generator to learn not only how to replicate pixel values ​​during training, but also how to capture and reproduce the details and realism of the image, improving the visual effect of the generated images.

[0087] Finally, through repeated iterative training and combined evaluation of perceptual loss, adversarial loss, and spectral consistency loss, the model parameters are continuously optimized so that the quality of the generated sketch image meets the expectations. After setting appropriate stopping conditions (such as the loss no longer decreases significantly or the performance on the validation set meets expectations), the training process ends. At this point, the generator network becomes the final face sketch synthesis model, which can convert new facial images into sketches to meet practical application needs.

[0088] See also Figure 2 The present invention further provides a face sketch synthesis system based on perceptual loss, the synthesis system is used to execute the above-mentioned face sketch synthesis method based on perceptual loss, comprising: A data acquisition module is used to obtain a facial image and a corresponding sketch image of an existing human face, wherein the facial image and the corresponding sketch image are mapped one by one, and face detection and alignment preprocessing are performed to ensure that the facial image and the sketch image have consistent facial features and the facial features are aligned one by one; The feature extraction module is used to extract features from the aligned facial images using the VGG19 network. The images are converted into feature maps through convolution and nonlinear activation functions layer by layer. In the features extracted at each layer, high-pass filters and low-pass filters are used to separate high-frequency detail features and low-frequency global features, and they are processed separately to form high-frequency feature maps and low-frequency feature maps. The feature fusion module is used to input the high-frequency and low-frequency feature maps into the NSRM module, and realize the weighted fusion of cross-resolution features through feature expansion and attention weight calculation. The feature expansion copies the original feature map to the batch size so that it can be compared with the current batch samples. The attention mechanism calculates the correlation weight between each feature map to realize the weighted superposition of features and form a unified fusion feature. The image synthesis module is used to input the fused features into the generator network. The generator network processes the feature map through the mirrored VGG decoder. The decoder gradually restores the image space structure through multiple layers of inverse convolution, maps the abstract feature vector back to the pixel space, and generates a preliminary synthetic sketch image through layer-by-layer upsampling and convolution operations; The loss processing module is used to input the generated sketch image and the corresponding real sketch image into the discriminator, extract features from the images through the convolution layer, calculate the authenticity score and adversarial loss of each pair of images, and calculate the perceptual loss and spectral consistency loss between the generated image and the real image; The training analysis module is used to update and adjust the parameters of the generator network and the discriminator based on the loss calculation results using the back propagation algorithm, repeat the iterative training, set the stop iteration conditions according to the loss calculation results, and obtain the face sketch synthesis model.

[0089] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0090] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0091] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0092] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A face sketch synthesis method based on perceptual loss, characterized in that: The specific steps include: Step 1: Obtain an existing facial image and a corresponding sketch image of a human face, map the facial image and the corresponding sketch image one by one, and perform facial detection and alignment preprocessing to ensure that the facial image and the sketch image have consistent facial features and the facial features are aligned one by one; Step 2: Use the VGG19 network to extract features from the aligned facial images. Through convolution and nonlinear activation functions layer by layer, the images are converted into feature maps. In the features extracted at each layer, high-pass filters and low-pass filters are used to separate high-frequency detail features and low-frequency global features, and they are processed separately to form high-frequency feature maps and low-frequency feature maps; Step 3: Input the high-frequency and low-frequency feature maps into the NSRM module, and realize the weighted fusion of cross-resolution features through feature expansion and attention weight calculation. Feature expansion copies the original feature map to the batch size so that it can be compared with the current batch samples. The attention mechanism calculates the correlation weight between each feature map to realize the weighted superposition of features and form a unified fusion feature. Step 4: Input the fused features into the generator network. The generator network processes the feature map through the mirrored VGG decoder. The decoder gradually restores the image space structure through multiple layers of inverse convolution, maps the abstract feature vector back to the pixel space, and generates a preliminary synthetic sketch image through layer-by-layer upsampling and convolution operations. Step 5: Input the generated sketch image and the corresponding real sketch image into the discriminator, extract features from the image through the convolution layer, calculate the authenticity score and adversarial loss for each pair of images, and calculate the perceptual loss and spectral consistency loss between the generated image and the real image; Step 6: Based on the loss calculation results, use the back propagation algorithm to update and adjust the parameters of the generator network and the discriminator, repeat the iterative training, set the stop iteration conditions according to the loss calculation results, and obtain the face sketch synthesis model.

2. The method for synthesizing face sketches based on perceptual loss according to claim 1, characterized in that: The specific logic for face detection and alignment preprocessing is: The MTCNN network is used to detect the facial image and the corresponding sketch image, and the facial feature points are extracted. The facial feature points include the center of the two eyes and the position of the nose tip. The distance between the eyes and the distance between the eyes and the nose are calculated to obtain the rotation parameter, the scaling parameter and the translation amount. The formula for calculating the distance between the eyes is: in, Represent the corresponding inter-eye distances in the facial image and the sketch image, respectively. Respectively represent the center coordinates of the left and right eye feature points in the facial image, Respectively represent the center coordinates of the left eye and right eye feature points in the sketch image; The formula for calculating the eye-nose distance is: in, Respectively represent the nose tip position and the eye-nose distance between the left eye and the right eye in the facial image, They represent the nose tip position and the eye-nose distance between the left eye and the right eye in the sketch image, respectively. Respectively represent the coordinates of the feature points of the nose tip in the facial image and the sketch image; The formula for obtaining the rotation parameters, scaling parameters, and translation is: in, denote the scaling factor and rotation angle respectively, Represent the rotation parameter and scaling parameter respectively, Respectively Direction and The amount of translation in the direction; Based on the affine transformation matrix, all pixels in the original facial image are transformed and the aligned facial image is generated. The formula is: in, They represent the coordinates of the pixel points in the facial image and the coordinates of the pixel points in the aligned facial image respectively.

3. The method for synthesizing face sketches based on perceptual loss according to claim 2, characterized in that: The logic for face detection and alignment preprocessing also includes adjusting all aligned face images and the corresponding sketch images to the same size, which is 512*512 pixels.

4. The method for synthesizing face sketches based on perceptual loss according to claim 1, characterized in that: Use the VGG19 network to extract features from the aligned facial images, and convert the images into feature maps layer by layer through convolution and nonlinear activation functions: The VGG19 network used consists of 16 convolutional layers with a 3x3 convolution kernel. The ReLU activation function is applied after each convolution layer to introduce nonlinearity. The 5 maximum pooling layers use a 2x2 pooling window for downsampling. Each pooling layer uses a 2x2 window with a step size of 2, and the maximum pooling layers are respectively after the 2nd, 4th, 7th, 13th and 16th convolutional layers; The first two convolutional layers and the maximum pooling layer behind them capture the facial contour, straight line of the bridge of the nose, outer edge of the lips and edge of the eyes of the aligned facial image, as well as the shadows caused by lighting changes. The second and third convolutional layers and the maximum pooling layer behind them identify the texture details on the skin, as well as the round features of the eyes and nostrils, the thickness and shape of the lips. The fourth to seventh convolutional layers and the maximum pooling layer behind them capture the internal structure of the eyes, including the boundary between the white of the eye and the iris, the curve of the eyelid, and the three-dimensional shape of the bridge of the nose and the opening and closing shape of the mouth. The eighth to twelfth convolutional layers and the maximum pooling layer behind them identify the shape of the face, including the contours of the cheeks and the shape of the chin, and identify the unique facial landmarks, including the location and shape of wrinkles and moles. The twelfth to sixteenth convolutional layers and the maximum pooling layer behind them identify the shape of the eyes and the shape of the chin and cheekbones, and identify and distinguish the facial feature combinations of different individuals, including face shape and facial features. The output feature map of the final maximum pooling layer is the feature map that finally transforms the image.

5. The method for synthesizing face sketches based on perceptual loss according to claim 4, characterized in that: When forming high-frequency feature maps and low-frequency feature maps, for each feature map generated by the convolution layer, a 3x3 Laplacian kernel is used as a high-pass filter to extract high-frequency information and obtain a high-frequency feature map. The formula is: in, Respectively represent the VGG19 network The feature map extracted by the convolutional layer and the high-frequency feature map after high-pass filtering, Represents the convolution operation; Use a 3x3 Gaussian kernel as a low-pass filter to extract low-frequency information and obtain a low-frequency feature map. The formula is: in, Indicates the VGG19 network The low-pass filtered low-frequency feature map extracted by the convolutional layer. The high-frequency feature map and low-frequency feature map formed in step 3 are the high-frequency feature map and low-frequency feature map output by the last maximum pooling layer.

6. The method for synthesizing face sketches based on perceptual loss according to claim 5, characterized in that: The method for forming a unified fusion feature in step 4 is: High-frequency feature map formed in step 3 And low frequency feature map , use the repeat function to expand it to the batch size , the formula based on is: in, Represent high-frequency feature maps And low frequency feature map After expansion Batch size high-frequency extended feature map and low-frequency extended feature map; The correlation between high-frequency and low-frequency features is calculated using the inner product of the feature map, based on the formula: in, is located in the high-frequency extended feature map High frequency feature map of and the position in the low-frequency extended feature map is Low frequency feature map of The relevance of represents the inner product of the two; For each set of feature vectors , calculate their similarity As a correlation matrix , and: in, represents the total number of all positions in the high-frequency feature map, Represents the total number of all positions in the low-frequency feature map; Use the Softmax function to transform the correlation matrix Normalize each row of to generate a correlation matrix The attention weight of each element in ; Based on the attention weight, the high-frequency and low-frequency features are fused. The attention weight is applied to the high-frequency features and superimposed on the low-frequency features for weighted fusion. The formula is: in, Indicates at location The fused feature vector is Indicates that the position in the high-frequency extended feature map is High frequency feature map of and the position in the low-frequency extended feature map is Low frequency feature map of The attention weights between Representation and location All related locations A collection of; The feature vectors for all positions Combined together to form a complete fusion feature map .

7. The method for synthesizing face sketches based on perceptual loss according to claim 6, characterized in that: The specific logic for generating a preliminary synthetic sketch image is: The fused feature map Input the generator network and fusion feature map conduct Convolution operation is used to adjust the number of channels: The decoder uses the mirror structure of VGG and gradually restores the image space through deconvolution and upsampling. In the mirror structure of VGG, the processing of each layer includes deconvolution, ReLU activation and upsampling, specifically: Each layer uses a transposed convolution to increase the resolution of the feature map, applies a ReLU activation function to provide non-linear features, and then upsamples the feature map, usually using bilinear interpolation to increase the resolution: Repeat deconvolution, activation and upsampling for each layer of feature maps until the size of the corresponding sketch image is reached; The processed feature map is restored to the image space, and the image generation is completed through transposed convolution and Tanh activation function to obtain a preliminary synthetic sketch image.

8. The method for synthesizing face sketches based on perceptual loss according to claim 1, characterized in that: The method used to calculate the authenticity score, adversarial loss, perceptual loss, and spectral consistency loss for each pair of images is: The generated synthetic sketch image and the corresponding real sketch image The input is sent to the discriminator network, and the local features of the image are extracted through the continuous convolutional layers in the discriminator network. Each convolutional layer passes through The convolution kernel, batch normalization and LeakyReLU activation function process the input feature map, and the final convolution layer outputs a high-dimensional feature map and ,in To synthesize sketch images The high-dimensional feature map generated by the convolution layer, A high-dimensional feature map generated by a convolutional layer for a real sketch image; The feature map and use The convolution layer is processed, and the Sigmoid function is used to obtain the authenticity scores of the synthetic sketch image and the real sketch image. The formula is: in, denote the authenticity scores of the synthetic sketch image and the real sketch image, respectively. and Represents the feature maps and use The value after the convolution layer is processed. Represents the Sigmoid function; The formulas for calculating adversarial loss and perceptual loss are: in, denote the adversarial loss and perceptual loss between the synthetic sketch image and the real sketch image, respectively. Represent the synthetic sketch images and the corresponding real sketch image Input to the discriminator network, The feature map output by the convolutional layer, represents the Euclidean distance between the two; The formula for calculating spectral consistency loss is: in, represents the spectral consistency loss between the synthetic sketch image and the real sketch image, and represents Fourier transform, which is used to convert the image into the frequency domain, represents the Euclidean distance.

9. The method for synthesizing face sketches based on perceptual loss according to claim 1, characterized in that: Based on the loss calculation results, the formula for updating and adjusting the parameters of the generator network and the discriminator using the back propagation algorithm is: in, Represent the parameter sets of the generator and discriminator respectively. The parameter sets include the weights of the connections between each layer of neurons in the generator and the discriminator and the bias of the neurons in each layer. denote the learning rates of the generator and the discriminator respectively, Represent the gradients of the generator and the discriminator respectively; The gradients of the generator and discriminator use the chain rule to propagate the gradient back from the output to the input. The calculation formula is: in, represents the total loss, and: in, denote the adversarial loss and perceptual loss between the synthetic sketch image and the real sketch image, respectively. represents the spectral consistency loss between the synthetic sketch image and the real sketch image, denote the adversarial loss weight, the perceptual loss weight, and the consistency loss weight, respectively, and All are not less than 0; When setting the conditions for stopping iteration, set a stopping threshold. If the total loss is less than the stopping threshold, the iterative training will be stopped.

10. A face sketch synthesis system based on perceptual loss, characterized by: The synthesis system is used to execute the face sketch synthesis method based on perceptual loss according to any one of claims 1 to 9, comprising: A data acquisition module is used to obtain a facial image and a corresponding sketch image of an existing human face, wherein the facial image and the corresponding sketch image are mapped one by one, and face detection and alignment preprocessing are performed to ensure that the facial image and the sketch image have consistent facial features and the facial features are aligned one by one; The feature extraction module is used to extract features from the aligned facial images using the VGG19 network. The images are converted into feature maps through convolution and nonlinear activation functions layer by layer. In the features extracted at each layer, high-pass filters and low-pass filters are used to separate high-frequency detail features and low-frequency global features, and they are processed separately to form high-frequency feature maps and low-frequency feature maps. The feature fusion module is used to input the high-frequency and low-frequency feature maps into the NSRM module, and realize the weighted fusion of cross-resolution features through feature expansion and attention weight calculation. The feature expansion copies the original feature map to the batch size so that it can be compared with the current batch samples. The attention mechanism calculates the correlation weight between each feature map to realize the weighted superposition of features and form a unified fusion feature. The image synthesis module is used to input the fused features into the generator network. The generator network processes the feature map through the mirrored VGG decoder. The decoder gradually restores the image space structure through multiple layers of inverse convolution, maps the abstract feature vector back to the pixel space, and generates a preliminary synthetic sketch image through layer-by-layer upsampling and convolution operations; The loss processing module is used to input the generated sketch image and the corresponding real sketch image into the discriminator, extract features from the images through the convolution layer, calculate the authenticity score and adversarial loss of each pair of images, and calculate the perceptual loss and spectral consistency loss between the generated image and the real image; The training analysis module is used to update and adjust the parameters of the generator network and the discriminator based on the loss calculation results using the back propagation algorithm, repeat the iterative training, set the stop iteration conditions according to the loss calculation results, and obtain the face sketch synthesis model.

Images