Face Sketch Image Generation Method and System Based on Spatial Adaptive Attention

Abstract

The present invention proposes a method and system for generating face sketch images based on spatial adaptive attention. The method includes: obtaining content features and style features from face photo images and face sketch images; respectively inputting the content features and style features into a content branch and a style branch to obtain deep content features and deep style features; using a spatial attention mechanism to adaptively weight and fuse the deep content features and deep style features to obtain adaptively weighted features; using the adaptively weighted features to obtain a reconstructed face sketch image; using the reconstructed face sketch image to optimize the model to obtain an optimized model; and using the optimized model to obtain face sketch images. By adaptively aggregating row vector and column vector information at the multi-scale semantic level, the present invention not only retains key facial details such as facial feature contours and hair strand textures, but also enhances the global structural consistency of the generated images.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image style transfer, and particularly relates to a method and system for generating face sketch images based on spatial adaptive attention. Background Art

[0002] The technology of generating face sketch images has shown important application value in the fields of digital entertainment, judicial criminal investigation, and medical diagnosis by converting face photo images into structured sketch images. In the field of digital entertainment, this technology can quickly generate personalized artistic images and lower the creation threshold for users; in judicial criminal investigation, it solves the matching problem between low-quality surveillance images and database photos through cross-modal matching and improves the efficiency of suspect identification; in medical diagnosis, the sketch form can enhance the visual expression of pathological features and assist doctors in accurate analysis. In the prior art, the research methods for face sketch generation are mainly divided into two categories: one is the traditional method, such as the edge detection technology based on filters; the other is the deep learning method, including the method based on convolutional neural network (CNN), the method based on Transformer, and the method based on diffusion model.

[0003] Traditional methods mostly rely on manually designed feature extraction and edge detection algorithms, which have the advantages of simple implementation and high computational efficiency, but have obvious deficiencies in dealing with complex backgrounds, maintaining edge continuity, and retaining detail information. Summary of the Invention

[0004] In view of the above situation, the main purpose of the present invention is to propose a method and system for generating face sketch images based on spatial adaptive attention to solve the above technical problems.

[0005] The present invention proposes a method for generating face sketch images based on spatial adaptive attention, and the method includes the following steps:

[0006] Step 1: After inputting the face photo image and the face sketch image into the face sketch image generation network, both are successively subjected to cropping and blocking and convolutional processing to respectively obtain the initial content feature and the initial style feature, and the initial content feature and the initial style feature are spliced with the corresponding learnable position embeddings to obtain the content feature and the style feature;

[0007] Step 2: Perform a cascaded max pooling operation and a large kernel convolution extraction operation on the content feature to obtain the content feature output after large kernel convolution processing;

[0008] Perform a multi-level average pooling strategy and a small kernel convolution extraction operation on the style feature to obtain the style feature output after small kernel convolution processing;

[0009] Perform spatial normalization on the content features output after large-kernel convolution processing and the style features output after small-kernel convolution processing to obtain attention weights;

[0010] Using the attention weights, spatially align the attention map with the original feature map through bilinear interpolation and obtain the spatially aligned attention weights;

[0011] Using the spatially aligned attention weights, dynamically calibrate the output content features and output style features in a weighted manner to obtain depth features;

[0012] Step 3: Use the spatial attention mechanism to adaptively weight and fuse the depth content features and depth style features to obtain the adaptively weighted features. At the same time, perform a linear projection operation on the depth style features alone to obtain the aligned style features;

[0013] Using the adaptively weighted features and the aligned style features, process them through the gating interaction mechanism in the row and column dimensions to obtain the fused features;

[0014] Step 4: Process the fused features through continuous upsampling decoding to obtain the reconstructed face sketch image;

[0015] Step 5: Use the reconstructed face sketch image, face photo image, and face sketch image to construct a loss function;

[0016] Optimize the face sketch image generation network in combination with the loss function to obtain the trained face sketch image generation network;

[0017] Input the face photo image and face sketch image into the trained face sketch image generation network to generate a face sketch image.

[0018] The present invention also proposes a face sketch image generation system based on spatial adaptive attention, and the system includes:

[0019] A feature extraction module for:

[0020] Input the face photo image and face sketch image into the face sketch image generation network, and both are sequentially subjected to cropping and blocking as well as convolution processing to respectively obtain initial content features and initial style features. Concatenate the initial content features and initial style features with the corresponding learnable position embeddings to obtain content features and style features;

[0021] A dual-modal feature decoupling module for:

[0022] Perform cascade max pooling operations and large-kernel convolution extraction operations on the content features to obtain the content features output after large-kernel convolution processing;

[0023] Adopt a multi-level average pooling strategy and small kernel convolution extraction operation for style features to obtain the style features output after small kernel convolution processing;

[0024] Perform spatial normalization on the content features output after large kernel convolution processing and the style features output after small kernel convolution processing to obtain attention weights;

[0025] Utilize the attention weights to spatially align the attention map with the original feature map through bilinear interpolation and obtain the attention weights after spatial alignment;

[0026] Utilize the attention weights after spatial alignment to dynamically calibrate the output content features and output style features in a weighted manner to obtain depth features;

[0027] The row-column vector dynamic fusion module is used for:

[0028] Utilize the spatial attention mechanism to adaptively weight and fuse the depth content features and depth style features to obtain adaptively weighted features. At the same time, perform a linear projection operation on the depth style features alone to obtain the aligned style features;

[0029] Utilize the adaptively weighted features and the aligned style features to process through the gating interaction mechanism in the row and column dimensions to obtain fused features;

[0030] The upsampling decoding module is used for:

[0031] Perform continuous upsampling decoding processing on the fused features to obtain the reconstructed face sketch image;

[0032] The training module is used for:

[0033] Utilize the reconstructed face sketch image, face photo image, and face sketch image to construct a loss function;

[0034] Combine the loss function to optimize the face sketch image generation network to obtain the trained face sketch image generation network;

[0035] The result output module is used for:

[0036] Input the face photo image and face sketch image into the trained face sketch image generation network to generate a face sketch painting image.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] 1. By integrating the dual-modal feature decoupling, dynamic feature fusion, and multi-scale adversarial optimization mechanisms, the present invention systematically solves the problems of structural distortion and texture blur existing in traditional cross-modal image generation, and significantly improves the clarity and realism of the generated sketch images;

[0039] 2. The present invention explicitly separates the semantic content and sketch style features of face images through a parallel branch structure, and introduces a spatial adaptive attention mechanism to dynamically align the cross-modal feature distributions, effectively alleviating the edge artifacts and information interference caused by modal differences and enhancing the edge retention ability.

[0040] 3. The present invention adaptively aggregates the row vector and column vector information at the multi-scale semantic level, not only retaining the key facial details such as the facial feature contours and hair textures, but also enhancing the global structural consistency of the generated images.

[0041] 4. The present invention adopts a combination of multi-layer reflection filling, convolution and upsampling to gradually restore the image resolution, effectively retaining the edge and texture details, and enhancing the structure restoration ability and the detail quality of the generated images.

[0042] 5. The present invention constructs a multi-granularity adversarial supervision mechanism by parallelly deploying a global discriminator and a local discriminator. While strengthening the authenticity of image details, it maintains the reasonable proportion and consistency of the facial contour structure, comprehensively enhancing the perceptual quality of the generated images.

[0043] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be understood through the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flowchart of the face sketch image generation method based on spatial adaptive attention proposed by the present invention;

[0045] Figure 2 is a schematic diagram of the dynamic fusion of column and row vectors of the face sketch image generation method based on spatial adaptive attention proposed by the present invention;

[0046] Figure 3 is a schematic diagram of the overall framework of the face sketch image generation system based on spatial adaptive attention proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

[0048] These and other aspects of the embodiments of the present invention will be clear with reference to the following description and drawings. In these descriptions and drawings, some specific embodiments of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0049] Please refer to Figure 1 , an embodiment of the present invention provides a method for generating a face sketch image based on spatial adaptive attention, and the method includes the following steps:

[0050] Step 1: After inputting the face photo image and the face sketch image into the face sketch image generation network, both are successively subjected to cropping and partitioning and convolutional processing to respectively obtain initial content features and initial style features, and the initial content features and initial style features are concatenated with corresponding learnable position embeddings to obtain content features and style features;

[0051] Furthermore, in this step, the specific processing steps for the face photo image and the face sketch image are as follows:

[0052] Crop the input face photo image and face sketch image into images with a size of 256×256 and a channel number of 3;

[0053] Divide the cropped image into multiple image blocks with a size of 8×8;

[0054] Perform a convolution operation with a convolution kernel size of 7×7 on each image block, and then embed it into a 512-dimensional high-dimensional feature space;

[0055] Finally, assign a learnable position embedding to each image block, and the position embedding is a 256-dimensional vector.

[0056] Step 2: Perform a cascaded max pooling operation and a large kernel convolution extraction operation on the content features to obtain the content features output after the large kernel convolution processing;

[0057] Perform a multi-level average pooling strategy and a small kernel convolution extraction operation on the style features to obtain the style features output after the small kernel convolution processing;

[0058] Perform spatial normalization processing on the content features output after the large kernel convolution processing and the style features output after the small kernel convolution processing to obtain attention weights;

[0059] Use the attention weights to spatially align the attention map with the original feature map through bilinear interpolation, and obtain the spatially aligned attention weights;

[0060] Use the spatially aligned attention weights to dynamically calibrate the output content features and output style features in a weighted manner to obtain depth features;

[0061] In step 2, a cascaded max pooling operation and a large kernel convolution extraction operation are performed on the content features to obtain the content features output after large kernel convolution. The specific steps are as follows:

[0062] Perform a cascaded max pooling operation on the content features to obtain the output content features. The relationship existing in the corresponding process is:

[0063] ;

[0064] Among them, represents the output content features, represents the max pooling process with a pooling kernel size of 2×2, represents the max pooling process with a pooling kernel size of 3×3, represents the content features;

[0065] Use alternately stacked 13×13 large kernel convolutions to process the output content features to obtain the content features output after large kernel convolution. The relationship existing in the corresponding process is:

[0066] ;

[0067] Among them, represents the content features output after large kernel convolution, represents the ReLU activation function, represents the process using alternately stacked 13×13 large kernel convolutions;

[0068] Perform a multi-level average pooling strategy and a small kernel convolution extraction operation on the style features to obtain the style features output after small kernel convolution. The specific steps are as follows:

[0069] Perform a multi-level average pooling strategy on the style features to obtain the output style features. The relationship existing in the corresponding process is:

[0070] ;

[0071] Among them, represents the output style features, represents the average pooling process with a pooling kernel size of 3×3, represents the average pooling process with a pooling kernel size of 5×5, represents the style features;

[0072] Use small kernel convolutions to replenish the local structure and detailed texture of the output style features to obtain the style features output after small kernel convolution. The relationship existing in the corresponding process is:

[0073] ;

[0074] Among them, represents the style feature output after small kernel convolution processing, represents processing using alternately stacked 3×3 small kernel convolutions;

[0075] Perform spatial normalization on the content feature output after large kernel convolution processing and the style feature output after small kernel convolution processing to obtain attention weights. The relational expression for the corresponding process is:

[0076] ;

[0077] Among them, represents the attention weight of the content feature map at position , represents the attention weight of the style feature map at position ; represents the content feature output after large kernel convolution processing at position ; represents the style feature output after small kernel convolution processing at position ; represents a small constant to prevent the denominator from being zero;

[0078] Using the attention weights, spatially align the attention map with the original feature map through bilinear interpolation and obtain the attention weights after spatial alignment. The relational expression for the corresponding process is:

[0079] ;

[0080] Among them, represents the content attention weight after spatial alignment, represents the style attention weight after spatial alignment, represents the attention weight of the content feature map, represents the attention weight of the style feature map, represents processing through a hyperbolic linear interpolation function, represents specifying the length and width after upsampling and using the attention weights after spatial alignment to dynamically calibrate the output content feature and the output style feature in a weighted manner to obtain depth features. The relational expression for the corresponding process is:

[0081] ;

[0082] Among them, represents the depth content feature, represents the depth style feature, represents element-wise multiplication;

[0083] Furthermore, in this step, a collaborative optimization framework for cross-modal feature decoupling is constructed through heterogeneous modality-driven feature extraction and dynamic spatial attention calibration to achieve the separation and alignment of face content features and sketch style features.

[0084] Step 3: Use the spatial attention mechanism to adaptively weight and fuse the depth content features and depth style features to obtain the adaptively weighted features. At the same time, perform a linear projection operation on the depth style features alone to obtain the aligned style features.

[0085] Use the adaptively weighted features and the aligned style features, and process them through the gated interaction mechanism in the row and column dimensions to obtain the fused features.

[0086] Please refer to Figure 2 , in Step 3, use the spatial attention mechanism to adaptively weight and fuse the depth content features and depth style features to obtain the adaptively weighted features. At the same time, perform a linear projection operation on the depth style features alone to obtain the aligned style features. The relational expressions for the corresponding process are:

[0087] ;

[0088] Among them, represents the adaptively weighted features, represents the aligned style features, represents the linear projection operation, represents the self-attention operation, represents the channel dimension concatenation operation;

[0089] Use the adaptively weighted features and the aligned style features, and through the processing of the gated interaction mechanism in the row and column dimensions, obtain the fused features. The specific steps are as follows:

[0090] Construct multi-norm attention branches for the adaptively weighted features and the aligned style features along the row and column vector directions respectively to obtain the row vector attention scores of the adaptively weighted features, the column vector attention scores of the adaptively weighted features, the row vector attention scores of the aligned style features, and the column vector attention scores of the aligned style features. The relational expressions for the corresponding process are:

[0091] ;

[0092] Among them, represents the total number of rows or columns of the features, represents the row vector attention score of the adaptively weighted features, represents the column vector attention score of the adaptively weighted features, represents the row vector attention score of the aligned style features, The column vector attention score representing the aligned style features Indicates being processed by the L1 norm Indicates being processed by the L2 norm Represents the L1 norm Represents the L2 norm Represents the maximum norm;

[0093] Using the exponential enhanced normalization strategy, nonlinearly amplify and probabilistically constrain the row and column vector attention scores of the features to respectively obtain the row vector attention score of the normalized adaptive weighted features, the column vector attention score of the normalized adaptive weighted features, the row vector attention score of the normalized aligned style features, and the column vector attention score of the normalized adaptive weighted features. The relational expressions existing in the corresponding process are:

[0094] ;

[0095] Among them, Indicates being processed by the exponential function Represents the row vector attention score of the normalized adaptive weighted features Represents the column vector attention score of the normalized adaptive weighted features Represents the row vector attention score of the normalized aligned style features Represents the column vector attention score of the normalized aligned style features;

[0096] Using the row and column vector attention scores of the normalized features to perform weighted fusion on the adaptive weighted features and the aligned style features to respectively obtain the row vector fusion features and the column vector fusion features. The relational expressions existing in the corresponding process are:

[0097] ;

[0098] Among them, Represents the row vector fusion features Represents the column vector fusion features;

[0099] Fuse the row vector fusion features and the column vector fusion features to obtain the fusion features. The relational expressions existing in the corresponding process are:

[0100] ;

[0101] Among them, Represents the fusion features.

[0102] Step 4, Process the fusion features through continuous upsampling decoding to obtain the reconstructed face sketch image;

[0103] In this step, the processing of the fused features is as follows:

[0104] Perform reflection padding on the fused features to preserve edge information;

[0105] Use a convolutional layer to reduce the number of channels from 512 dimensions to 256 dimensions, and enhance the non-linear expression through the ReLU activation function;

[0106] Use the nearest neighbor interpolation method for the first upsampling to expand the spatial size;

[0107] Further enhance the feature expression ability through multiple repeated reflection padding and convolution operations, which includes three consecutive convolutional modules, and each module consists of reflection padding, 3×3 convolution, and the ReLU activation function;

[0108] Reduce the channel dimension from 256 to 128, and perform the second upsampling operation;

[0109] Repeat the above operations to further reduce the channel dimension from 128 to 64, and perform the third upsampling;

[0110] Finally, map the channel dimension from 64 to 3 channels of the output image through convolution to obtain the reconstructed face sketch image.

[0111] Step 5: Construct a loss function using the reconstructed face sketch image, face photo image, and face sketch image;

[0112] Optimize the face sketch image generation network in combination with the loss function to obtain the trained face sketch image generation network;

[0113] Input the face photo image and face sketch image into the trained face sketch image generation network to generate a face sketch painting image;

[0114] In step 5, construct a loss function using the reconstructed face sketch image, face photo image, and face sketch image. Among them, the loss function includes a content loss function, a style loss function, a detail loss function, an identity loss function, and an adversarial loss function. The content loss function:

[0115] ;

[0116] Among them, represents the content loss function, represents the reconstructed face sketch image, represents the face photo image, represents the feature extraction operation using a deep convolutional neural network;

[0117] The style loss function:

[0118] ;

[0119] Among them, represents the style loss function, represents the face sketch image, represents calculating the channel mean of the feature map, represents calculating the standard deviation of the feature map;

[0120] Detail loss function:

[0121] ;

[0122] Among them, represents the detail loss function, represents the mask of the edge region, represents being processed by a Gaussian filter, represents through the Laplacian operator;

[0123] Identity loss function:

[0124] ;

[0125] Among them, and both represent the identity loss function, after being extracted and processed by the th layer of the deep convolutional neural network;

[0126] Adversarial loss function:

[0127] ;

[0128] Among them, represents the adversarial loss function, represents the number of reconstructed face sketch images, represents the th discriminator, represents the face sketch image generated from the th face photo image;

[0129] Total loss function:

[0130] ;

[0131] Among them, represents the total loss function, represents the content loss weight, represents the style loss weight, represents the detail loss weight, and both represent the identity loss weight, Denotes the adversarial loss weight;

[0132] Furthermore, in this step, a generative adversarial network framework is adopted, and adversarial training is carried out through the model architectures of a generator and a multi-scale perception discriminator. The multi-scale perception discriminator includes the following steps:

[0133] The discriminator forms a multi-scale structure perception ability by introducing multiple discriminative sub-networks, and each sub-network acts on face image inputs of different scales respectively. Downsampling is performed through average pooling between each scale, thereby constructing a discriminative pyramid;

[0134] Each discriminative sub-network adopts a convolutional neural network based on the PatchGAN structure, which consists of multiple layers of convolutional layers, non-linear activation layers and normalization layers, including:

[0135] The initial convolutional layer maps the input image from the input channels to the base channel dimension, followed by a LeakyReLU activation function;

[0136] A multi-layer residual convolutional module, where each layer doubles the number of channels through convolutional operations, with a maximum of no more than 512 dimensions, and combines instance normalization (InstanceNorm) for feature regularization;

[0137] The last layer outputs a single-channel feature map for discriminating the authenticity of the image patch;

[0138] Supports outputting discriminative feature maps of intermediate layers for multi-level feature supervision, thereby enhancing the sensitivity to local texture and structural changes;

[0139] The network model parameters are initialized with a Gaussian distribution, and support GPU-accelerated deployment and multi-scale distributed parallel training;

[0140] The results of the discriminative sub-networks will be combined and used to provide discriminative feedback signals from different scales to the generator, thereby guiding it to generate a face sketch image with both overall structure and local texture authenticity.

[0141] Furthermore, calculate metrics such as SSIM between the generated face sketch image and the real sketch, and finally complete the quality evaluation of the face sketch image.

[0142] Please refer to Figure 3 , this embodiment of the present invention also provides a face sketch image generation system based on spatial adaptive attention. The system includes: <(

[0143] A feature extraction module for:

[0144] After inputting the face photo image and the face sketch image into the face sketch image generation network, they both go through cropping and chunking, as well as convolutional processing in sequence, respectively obtaining the initial content feature and the initial style feature. The initial content feature and the initial style feature are concatenated with the corresponding learnable position embeddings to obtain the content feature and the style feature;

[0145] The dual-modal feature decoupling module is used for:

[0146] Performing cascaded max pooling operation and large kernel convolution extraction operation on the content feature to obtain the content feature output after large kernel convolution processing;

[0147] Performing multi-level average pooling strategy and small kernel convolution extraction operation on the style feature to obtain the style feature output after small kernel convolution processing;

[0148] Performing spatial normalization processing on the content feature output after large kernel convolution processing and the style feature output after small kernel convolution processing to obtain the attention weight;

[0149] Using the attention weight, spatially aligning the attention map with the original feature map through bilinear interpolation, and obtaining the spatially aligned attention weight;

[0150] Using the spatially aligned attention weight to dynamically calibrate the output content feature and the output style feature in a weighted manner to obtain the depth feature;

[0151] The row-column vector dynamic fusion module is used for:

[0152] Using the spatial attention mechanism to adaptively weight and fuse the depth content feature and the depth style feature to obtain the adaptively weighted feature. At the same time, performing a linear projection operation on the depth style feature alone to obtain the aligned style feature;

[0153] Using the adaptively weighted feature and the aligned style feature to process through the gated interaction mechanism in the row and column dimensions to obtain the fusion feature;

[0154] The upsampling decoding module is used for:

[0155] Performing continuous upsampling decoding processing on the fusion feature to obtain the reconstructed face sketch image;

[0156] The training module is used for:

[0157] Using the reconstructed face sketch image, the face photo image, and the face sketch image to construct a loss function;

[0158] Combining the loss function to optimize the face sketch image generation network to obtain the trained face sketch image generation network;

[0159] A result output module, configured to:

[0160] Input the face photo image and the face sketch image into the trained face sketch image generation network to generate a face sketch image.

[0161] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following technologies well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logic functions on data signals, application specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0162] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0163] The above-described embodiments merely represent several implementation manners of the present invention. Their descriptions are relatively specific and detailed, but should not be construed as limiting the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be subject to the appended claims.

Claims

1. A method for generating face sketch images based on spatial adaptive attention, characterized in that, The method includes the following steps: Step 1: After inputting the face photo image and the face sketch image into the face sketch image generation network, both are successively subjected to cropping and partitioning as well as convolutional processing to respectively obtain the initial content feature and the initial style feature. The initial content feature and the initial style feature are concatenated with the corresponding learnable position embeddings to obtain the content feature and the style feature; Step 2: The cascade max pooling operation and the large kernel convolution extraction operation are adopted for the content feature to obtain the content feature output after the large kernel convolution processing; The multi-level average pooling strategy and the small kernel convolution extraction operation are adopted for the style feature to obtain the style feature output after the small kernel convolution processing; Spatial normalization processing is performed on the content feature output after the large kernel convolution processing and the style feature output after the small kernel convolution processing to obtain the attention weight; Using the attention weight, the attention map and the original feature map are spatially aligned through bilinear interpolation, and the spatially aligned attention weight is obtained; Using the spatially aligned attention weight, the output content feature and the output style feature are dynamically calibrated in a weighted manner to obtain the depth feature; Step 3: The spatial attention mechanism is used to adaptively weightedly fuse the depth content feature and the depth style feature to obtain the adaptively weighted feature. At the same time, a linear projection operation is separately performed on the depth style feature to obtain the aligned style feature; Using the adaptively weighted feature and the aligned style feature, the fused feature is obtained through the gating interaction mechanism processing in the row and column dimensions; Step 4: The fused feature is processed through continuous upsampling decoding to obtain the reconstructed face sketch image; Step 5: A loss function is constructed using the reconstructed face sketch image, the face photo image, and the face sketch image; The face sketch image generation network is optimized in combination with the loss function to obtain the trained face sketch image generation network; The face photo image and the face sketch image are input into the trained face sketch image generation network to generate a face sketch painting image.

2. The method for generating a face sketch image based on spatial adaptive attention according to claim 1, wherein In the said Step 2, the cascade max pooling operation and the large kernel convolution extraction operation are adopted for the content feature to obtain the content feature output after the large kernel convolution processing. The specific steps are as follows: The cascade max pooling operation is adopted for the content feature to obtain the output content feature. The relational expression existing in the corresponding process is: ； Among them, represents the output content feature, represents the maximum pooling process with a pooling kernel size of 2×2, represents the maximum pooling process with a pooling kernel size of 3×3, represents the content feature; The output content feature is processed using alternately stacked 13×13 large kernel convolutions to obtain the content feature output after the large kernel convolution processing. The relational expression existing in the corresponding process is: ； Among them, represents the content feature output after large kernel convolution processing, represents the ReLU activation function, represents the processing using alternately stacked 13×13 large kernel convolutions.

3. The method for generating a face sketch image based on spatial adaptive attention according to claim 2, wherein In the said Step 2, the multi-level average pooling strategy and the small kernel convolution extraction operation are adopted for the style feature to obtain the style feature output after the small kernel convolution processing. The specific steps are as follows: The multi-level average pooling strategy is adopted for the style feature to obtain the output style feature. The relational expression existing in the corresponding process is: ； Among them, represents the output style feature, represents average pooling processing with a pooling kernel size of 3×3, represents average pooling processing with a pooling kernel size of 5×5, represents the style feature; The local structure and detail texture of the output style feature are replenished using small kernel convolutions to obtain the style feature output after the small kernel convolution processing. The relational expression existing in the corresponding process is: ； Among them, represents the style feature output after small kernel convolution processing, represents the processing using alternately stacked 3×3 small kernel convolutions.

4. The method for generating a face sketch image based on spatial adaptive attention according to claim 3, wherein In step 2, spatial normalization is performed on the content features output after large kernel convolution processing and the style features output after small kernel convolution processing to obtain attention weights. The relational expression existing in the corresponding process is: ； Among them, represents the attention weight of the content feature map at the position . represents the attention weight of the style feature map at the position . represents the content feature output after large kernel convolution processing at the position . represents the style feature output after small kernel convolution processing at the position . represents a small constant to prevent the denominator from being zero.

5. The method for generating a face sketch image based on spatial adaptive attention according to claim 4, wherein, In step 2, using the attention weights, the attention map and the original feature map are spatially aligned through bilinear interpolation, and the attention weights after spatial alignment are obtained. The relational expression existing in the corresponding process is: ； Among them, represents the content attention weight after spatial alignment, represents the style attention weight after spatial alignment, represents the attention weight of the content feature map, represents the attention weight of the style feature map, represents being processed by the hyperbolic linear interpolation function, represents specifying the length and width after upsampling.

6. The method for generating a face sketch image based on spatial adaptive attention according to claim 5, wherein In step 2, using the attention weights after spatial alignment, the output content features and the output style features are dynamically calibrated in a weighted manner to obtain depth features. The relational expression existing in the corresponding process is: ； Among them, represents the depth content feature, represents the depth style feature, represents element-wise multiplication.

7. The method for generating a face sketch image based on spatial adaptive attention according to claim 6, wherein In step 3, using the spatial attention mechanism, adaptive weighted fusion is performed on the depth content features and the depth style features to obtain adaptive weighted features. At the same time, a linear projection operation is separately performed on the depth style features to obtain the aligned style features. The relational expression existing in the corresponding process is: ； Among them, represents the adaptive weighted feature, represents the aligned style feature, represents the linear projection operation, represents the self-attention operation, represents the channel dimension concatenation operation.

8. The method for generating a face sketch image based on spatial adaptive attention according to claim 7, wherein In step 3, using the adaptive weighted features and the aligned style features, through the gating interaction mechanism of row and column dimensions, fusion features are obtained. The specific steps are: Multi-norm attention branches are respectively constructed for the adaptive weighted features and the aligned style features along the row and column vector directions to respectively obtain the row vector attention scores of the adaptive weighted features, the column vector attention scores of the adaptive weighted features, the row vector attention scores of the aligned style features, and the column vector attention scores of the aligned style features. The relational expression existing in the corresponding process is: ； Among them, represents the total number of rows or columns of features, represents the row vector attention score of the adaptive weighted features, represents the column vector attention score of the adaptive weighted features, represents the row vector attention score of the aligned style features, represents the column vector attention score of the aligned style features, represents being processed by the L1 norm, represents being processed by the L2 norm, represents the L1 norm, represents the L2 norm, represents the maximum norm; Using the exponential enhancement normalization strategy, non-linear amplification and probability constraint are performed on the row and column vector attention scores of the features to respectively obtain the normalized row vector attention scores of the adaptive weighted features, the normalized column vector attention scores of the adaptive weighted features, the normalized row vector attention scores of the aligned style features, and the normalized column vector attention scores of the adaptive weighted features. The relational expression existing in the corresponding process is: ； Among them, indicates being processed by the exponential function, represents the row vector attention score of the normalized adaptive weighted features, represents the column vector attention score of the normalized adaptive weighted features, represents the row vector attention score of the normalized aligned style features, represents the column vector attention score of the normalized aligned style features; Using the normalized row and column vector attention scores of the features, weighted fusion is performed on the adaptive weighted features and the aligned style features to respectively obtain the row vector fusion features and the column vector fusion features. The relational expression existing in the corresponding process is: ； Among them, represents the fused feature of row vectors, represents the fused feature of column vectors; Fusion is performed on the row vector fusion features and the column vector fusion features to obtain fusion features. The relational expression existing in the corresponding process is: ; Among them, represents the fusion feature.

9. The method for generating a face sketch image based on spatial adaptive attention according to claim 8, wherein In step 5, a loss function is constructed using the reconstructed face sketch image, face photo image, and face sketch image. Among them, the loss function includes a content loss function, a style loss function, a detail loss function, an identity loss function, and an adversarial loss function. The expression of the content loss function is: ； Among them, represents the content loss function, represents the reconstructed face sketch image, represents the face photo image, represents the feature extraction operation performed using a deep convolutional neural network; The expression of the style loss function is: ； Among them, represents the style loss function, represents the face sketch image, represents calculating the channel mean of the feature map, represents calculating the standard deviation of the feature map; The expression of the detail loss function is: ； Among them, represents the detail loss function, represents the mask of the edge region, represents being processed by a Gaussian filter, represents through the Laplacian operator; The expression of the identity loss function is: ； Among them, and both represent the identity loss function, after being processed by the extraction of the th layer of the deep convolutional neural network; The expression of the adversarial loss function is: ； Among them, represents the adversarial loss function, represents the number of reconstructed face sketch images, represents the th discriminator, represents the face sketch image generated from the th face photo image; The expression of the total loss function is: ； Among them, represents the total loss function, represents the content loss weight, represents the style loss weight, represents the detail loss weight, and both represent the identity loss weight, represents the adversarial loss weight.

10. A face sketch image generation system based on spatial adaptive attention, characterized in that, The system applies the method for generating a face sketch image based on spatial adaptive attention according to any one of claims 1 to 9. The system includes: A feature extraction module for: After inputting the face photo image and the face sketch image into the face sketch image generation network, they both go through cropping and chunking, and convolutional processing in sequence, respectively obtaining the initial content feature and the initial style feature. The initial content feature and the initial style feature are concatenated with the corresponding learnable position embeddings to obtain the content feature and the style feature; The dual-modal feature decoupling module is used for: Performing a cascaded max pooling operation and a large kernel convolution extraction operation on the content feature to obtain the content feature output after large kernel convolution processing; Performing a multi-level average pooling strategy and a small kernel convolution extraction operation on the style feature to obtain the style feature output after small kernel convolution processing; Performing spatial normalization processing on the content feature output after large kernel convolution processing and the style feature output after small kernel convolution processing to obtain the attention weight; Using the attention weight, spatially aligning the attention map with the original feature map through bilinear interpolation, and obtaining the spatially aligned attention weight; Using the spatially aligned attention weight to dynamically calibrate the output content feature and the output style feature in a weighted manner to obtain the depth feature; The row-column vector dynamic fusion module is used for: Using the spatial attention mechanism to adaptively weight-fuse the depth content feature and the depth style feature to obtain the adaptively weighted feature. At the same time, performing a linear projection operation on the depth style feature alone to obtain the aligned style feature; Using the adaptively weighted feature and the aligned style feature, processing through the gated interaction mechanism in the row-column dimension to obtain the fusion feature; The upsampling decoding module is used for: Performing continuous upsampling decoding processing on the fusion feature to obtain the reconstructed face sketch image; The training module is used for: Constructing a loss function using the reconstructed face sketch image, the face photo image, and the face sketch image; Combining the loss function to optimize the face sketch image generation network to obtain the trained face sketch image generation network; The result output module is used for: Inputting the face photo image and the face sketch image into the trained face sketch image generation network to generate the face sketch painting image.

Images