A Method and System for Generating Face Sketches Based on Mamba and Wavelet Convolution

Abstract

This invention proposes a method and system for generating face sketches based on Mamba and wavelet convolution. The method includes: processing the image using an image processing module to obtain preliminary content features and preliminary style features; adding positional information to the preliminary content features to obtain preliminary content features with added positional information; inputting the preliminary content features with added positional information into a Mamba-Transformer encoder module for processing to obtain deep content features; inputting the preliminary style features into a style-enhanced wavelet convolution module for processing to obtain style-enhanced style features; inputting the style-enhanced style features into a Mamba-Transformer encoder module for processing to obtain deep style features; and inputting the deep content features and deep style features into a decoder module for processing to obtain a face sketch image. This invention can generate high-resolution, structurally accurate, and naturally styled face sketch images.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for generating human face sketches based on Mamba and wavelet convolution. Background Technology

[0002] Facial sketching image generation technology, by converting facial photographs into structured sketch images, has significant application value in fields such as digital entertainment, forensic investigation, and medical diagnosis. In digital entertainment, this technology can quickly generate personalized artistic images, lowering the barrier to user creation; in forensic investigation, cross-modal matching can solve the problem of matching low-quality surveillance images with database photos, improving the efficiency of suspect identification; in medical diagnosis, the sketch format can enhance the visual expression of pathological features, assisting doctors in making accurate analyses.

[0003] Existing methods for generating facial sketches mainly fall into two categories: traditional methods, such as filter-based edge detection techniques, and deep learning methods, including methods based on convolutional neural networks and Transformers. Traditional methods rely on hand-designed feature extraction and edge detection algorithms, which are simple to implement and computationally efficient, but have significant limitations in handling complex backgrounds, maintaining edge continuity, and preserving details.

[0004] With the development of deep learning, convolutional neural networks have been widely used due to their powerful ability to extract local features. Although they have achieved some success in detail restoration, convolutional neural networks still have shortcomings in global structure modeling due to their limited receptive field. Summary of the Invention

[0005] In view of the above situation, the main objective of this invention is to propose a face sketching generation method based on Mamba and wavelet convolution to solve the above-mentioned technical problems.

[0006] This invention proposes a face sketching generation method based on Mamba and wavelet convolution, the method comprising the following steps:

[0007] Step 1: Input the content image and style image into the image processing module, crop the content image and style image respectively to obtain cropped content image and style image; flatten the cropped content image and style image respectively and process them through linear mapping to obtain preliminary content features and preliminary style features; add position information to the preliminary content features to obtain preliminary content features with added position information.

[0008] Step 2: Input the preliminary content features after adding location information into the Mamba-Transformer encoder module for processing to obtain deep content features;

[0009] Step 3: Input the preliminary style features into the style-enhanced wavelet convolution module for processing to obtain style-enhanced style features;

[0010] Step 4: Input the style-enhanced style features into the Mamba-Transformer encoder module for processing to obtain deep style features;

[0011] Step 5: Input the depth content features and depth style features into the decoder module for processing to obtain a face sketch image.

[0012] This invention also proposes a face sketching generation system based on Mamba and wavelet convolution, the system comprising:

[0013] Image processing module, used for:

[0014] The content image and style image are cropped separately to obtain cropped content images and style images; the cropped content images and style images are flattened and then processed by linear mapping to obtain preliminary content features and preliminary style features; position information is added to the preliminary content features to obtain preliminary content features with added position information.

[0015] The Mamba-Transformer encoder module is used for:

[0016] The initial content features after incorporating location information are processed to obtain deep content features;

[0017] The initial stylistic features are processed to obtain enhanced stylistic features;

[0018] Style-enhanced wavelet convolution module, used for:

[0019] The style features enhanced by style are processed to obtain deep style features;

[0020] Decoder module, used for:

[0021] Deep content features and deep style features are processed to obtain a facial sketch image.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] 1. This invention systematically solves the structural distortion and texture blurring problems existing in traditional cross-modal image generation by integrating the Mamba-Transformer encoder, style-enhanced wavelet convolution and decoder module, and significantly improves the clarity, detail reproduction and realism of the generated sketch images.

[0024] 2. This invention achieves joint modeling of global dependencies and local details by setting up a visual Mamba module in parallel and a simplified variable-size window attention module, which improves the ability to express structural information and semantic features while ensuring modeling efficiency.

[0025] 3. By setting up style enhancement convolution modules and wavelet convolution modules in parallel, this invention can simultaneously capture multi-directional difference features and multi-frequency domain wavelet features, thereby enhancing texture expression and style consistency, and significantly improving the artistic sense and detail of the generated image.

[0026] 4. This invention achieves efficient fusion of cross-modal features and layer-by-layer spatial reconstruction by setting up the Transformer decoder module and the convolutional decoder module in parallel. This ensures the matching of global style and structure, and preserves detailed features during the progressive upsampling process. Ultimately, it can generate high-resolution, structurally accurate and style-natural facial sketch images. Attached Figure Description

[0027] Figure 1 The flowchart shows the face sketching generation method based on Mamba and wavelet convolution proposed in this invention.

[0028] Figure 2 This is a schematic diagram of the Mamba-Transformer encoder module of the face sketching generation method based on Mamba and wavelet convolution proposed in this invention.

[0029] Figure 3 This is a schematic diagram of the style-enhanced wavelet convolution module of the face sketching generation method based on Mamba and wavelet convolution proposed in this invention.

[0030] Figure 4 This is a schematic diagram of the quantitative comparison experiments conducted by the present invention on four different datasets with different models.

[0031] Figure 5 This is a schematic diagram of the framework of the face sketching generation system based on Mamba and wavelet convolution proposed in this invention. Detailed Implementation

[0032] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0033] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0034] Please see Figure 1 This invention proposes a face sketching generation method based on Mamba and wavelet convolution, which includes the following steps:

[0035] Step 1: Input the content image and style image into the image processing module. Crop the content image and style image respectively to obtain cropped content image and style image. Flatten the cropped content image and style image respectively, and then perform linear mapping to obtain preliminary content features and preliminary style features. Add positional information to the preliminary content features to obtain preliminary content features with added positional information.

[0036] In step 1, location information is added to the preliminary content features to obtain the preliminary content features after adding location information. The corresponding relationship in this process is as follows:

[0037] ;

[0038] in, This indicates the initial content characteristics after adding location information. Indicates preliminary content characteristics, Indicates position code, This indicates that the feature maps are added element by element.

[0039] Step 2: Input the preliminary content features after adding location information into the Mamba-Transformer encoder module for processing to obtain deep content features;

[0040] Please see Figure 3 In step 2, the preliminary content features after incorporating location information are input into the Mamba-Transformer encoder module for processing to obtain depth content features. The specific steps are as follows:

[0041] The Mamba-Transformer encoder module consists of a visual Mamba module and a simplified variable-size window attention module in parallel;

[0042] The initial content features after adding location information are processed through the state space model branch in the visual Mamba module to obtain the output features after state space branch processing.

[0043] The initial content features after adding location information are processed by the symmetric convolution branch in the visual Mamba module to obtain the output features after the symmetric convolution branch processing.

[0044] The output features processed by the state space branch are fused with the output features processed by the symmetric convolution branch to obtain the output features processed by the visual Mamba module.

[0045] The preliminary content features, after incorporating location information, are input into a simplified variable-size window attention module for processing to obtain the predicted window size and the predicted window offset.

[0046] The dynamic window features, key matrix, and value matrix are obtained by using the predicted window size, predicted window offset, and preliminary content features after adding position information;

[0047] The simplified variable-size window output features are obtained by utilizing dynamic window features, the key matrix, and the value matrix after processing by the attention module.

[0048] The output features processed by the visual Mamba module and the output features processed by the simplified variable-size window attention module are weighted and fused, and then processed by convolution to obtain the fused features.

[0049] The fused features are sequentially processed by a feedforward neural network and then by layer normalization to obtain deep content features.

[0050] The process of obtaining fusion features from preliminary content features can be expressed by the following relational formula:

[0051] ;

[0052] in, This indicates that the process is handled by stacking three consecutive Mamba-Transformer modules.

[0053] In the process of processing the initial content features after adding location information through the state space model branch in the visual Mamba module to obtain the output features after state space branch processing, the following relationship exists:

[0054] ;

[0055] in, This represents the output feature after state-space branching. This indicates that it is processed through a state-space model. This indicates that the process is performed using the SiLU activation function. This indicates processing via a one-dimensional convolutional layer. This indicates that the data has undergone linear mapping.

[0056] In this process, the state-space model branch is used to capture global features of long-range dependencies;

[0057] In the process of processing the initial content features after adding location information through the symmetric convolution branch in the visual Mamba module to obtain the output features after symmetric convolution branch processing, the following relationship exists:

[0058] ;

[0059] in, This represents the output features after processing by symmetric convolution branches;

[0060] In this process, symmetric convolution branches are used to mitigate information loss caused by sequential constraints and to supplement local features;

[0061] In the process of fusing the output features after state space branch processing with the output features after symmetric convolution branch processing to obtain the output features after visual Mamba module processing, the following relationship exists:

[0062] ;

[0063] in, This represents the output features after processing by the visual Mamba module. This indicates splicing along the channel dimension;

[0064] In the process of inputting the preliminary content features after incorporating location information into a simplified variable-size window attention module for processing to obtain the predicted window size and predicted window offset, the following relationship exists:

[0065] ;

[0066] in, Indicates the size of the prediction window. This represents the predicted window offset. This indicates that a convolution operation has been performed. This indicates that the device has been processed using an activation function with a negative slope. This indicates that global average pooling has been applied.

[0067] In this process, a simplified variable-size window attention module is used to capture local detail features of the features;

[0068] In the process of obtaining dynamic window features, key matrix, and value matrix using the predicted window size, predicted window offset, and preliminary content features after adding positional information, the following relationships exist:

[0069] ;

[0070] in, Indicates dynamic window characteristics, This indicates that a spatial Transformer operation has been performed. Represents the key matrix. Represents a value matrix, This represents a reshaping operation that flattens a feature map into a sequence.

[0071] In the process of obtaining the simplified variable-size window attention module output features using dynamic window features, key matrix, and value matrix, the following relationship exists:

[0072] ;

[0073] in, This represents the local key matrix obtained by window sampling. This represents the local value matrix obtained by window sampling. This indicates local attention computation. This represents the simplified output characteristics after processing by the variable-size window attention module. This indicates that the linear window attention process has been applied.

[0074] In the process of weightedly fusing the output features processed by the visual Mamba module and the output features processed by the simplified variable-size window attention module, and then performing convolution to obtain the fused features, the following relationship exists:

[0075] ;

[0076] in, Indicates fusion characteristics, and All of these represent trainable weights. This indicates that elements at corresponding positions are multiplied;

[0077] In this process, the Mamba-Transformer module performs weighted fusion of global and local features extracted by the visual Mamba module and the simplified variable-size window attention module through trainable weights, and then integrates the channels with a convolution kernel of size 1×1 to obtain fused features;

[0078] The process of sequentially processing the fused features using a feedforward neural network and then performing layer normalization to obtain deep content features involves the following relationship:

[0079] ;

[0080] in, Indicates depth content characteristics, Presentation layer normalization processing, This indicates that the data has been processed by a feedforward neural network.

[0081] In this process, the fused features are enhanced by a feedforward network and combined with layer normalization to further extract structural and semantic information and obtain deep content features.

[0082] Step 3: Input the preliminary style features into the style-enhanced wavelet convolution module for processing to obtain style-enhanced style features;

[0083] Please see Figure 2 In step 3, the preliminary style features are input into the style-enhanced wavelet convolution module for processing to obtain style-enhanced style features. The specific steps are as follows:

[0084] The style-enhanced wavelet convolution module consists of a style-enhanced convolution module and a wavelet convolution module in parallel.

[0085] Five differential convolutions in the style-enhanced wavelet convolution module are used to perform multi-directional feature extraction on the initial style features to obtain the output features of the summation path and the output features of the concatenation path.

[0086] The gating weights are obtained by using the output features of the summation path and the output features of the concatenation path, and then the output features of the style enhancement convolutional module are obtained by using the gating weights.

[0087] The initial style features are decomposed using a wavelet convolution module to obtain the first layer of low-frequency components and the first layer of high-frequency components.

[0088] The first low-frequency component is decomposed into wavelet components to obtain the second low-frequency component and the second high-frequency component.

[0089] The second-layer low-frequency component and the second-layer high-frequency component are subjected to the first-layer inverse wavelet reconstruction process to obtain the enhanced and recovered low-frequency component.

[0090] Based on the enhanced recovery of low-frequency components, first-layer high-frequency components, and first-layer low-frequency components, the output features of the wavelet convolution module are obtained through second-layer inverse wavelet reconstruction.

[0091] The output features of the style enhancement convolutional module and the output features of the wavelet convolutional module are subjected to channel attention processing and spatial attention weighted fusion processing to obtain enhanced style features;

[0092] The initial style features are linearly transformed to obtain the query vector of the matrix;

[0093] The initial style features and the enhanced style features are stacked and then a linear transformation is performed to obtain the key vector and value vector of the matrix.

[0094] Based on the query vector, key vector, and value vector of the matrix, attention weights are obtained through an attention mechanism.

[0095] The initial style features and the enhanced style features are weighted and summed using attention weights to obtain a weighted sum result. Then, the weighted sum result is subjected to tensor dimension compression to obtain the style features enhanced by style.

[0096] In the process of using five differentiable convolutions in the style-enhanced wavelet convolution module to perform multi-directional feature extraction on the initial style features to obtain the output features of the summation path and the output features of the concatenation path, the following relationship is included:

[0097] ;

[0098] in, This indicates the output characteristics of the summation path. Indicates initial stylistic characteristics, Represents convolution and indexing. The output features represent the concatenation path. This indicates that the process has undergone a second convolutional layer. This represents the output features of a regular convolution. This represents the output feature of the central difference convolution. This represents the output features of the angular difference convolution. This represents the output features of the horizontal difference convolution. This represents the output features of vertical difference convolution;

[0099] In the process of obtaining gating weights by utilizing the output features of the summation path and the concatenation path, and then using the gating weights to obtain the output features of the style enhancement convolutional module, the following relationship exists:

[0100] ;

[0101] in, Indicates the gating weight, This indicates that the element-wise sigmoid function has been applied. Representation style enhancement of the output features of the convolutional module;

[0102] In the process of using wavelet convolution modules to decompose the initial style features to obtain the first layer of low-frequency components and the first layer of high-frequency components, the following relationship exists:

[0103] ;

[0104] in, This represents the first layer of low-frequency components. , and Both represent the first layer of high-frequency components. This indicates that the signal has undergone convolution processing using the corresponding wavelet filter.

[0105] In this process, the initial style features are split into low-frequency and high-frequency components by wavelet decomposition to extract global and local information respectively, and the first layer of low-frequency components and the first layer of high-frequency components are obtained.

[0106] The process of performing wavelet decomposition on the first-level low-frequency components to obtain the second-level low-frequency components and the second-level high-frequency components can be summarized as follows:

[0107] ;

[0108] in, This indicates the low-frequency components of the second layer. , and Both represent the high-frequency components of the second layer. This indicates that the process has undergone a second wavelet decomposition.

[0109] In this process, the low-frequency components obtained from the first layer of decomposition are further decomposed into wavelet components to obtain low-frequency and high-frequency components at a smaller scale, so as to obtain the second layer of low-frequency components and the second layer of high-frequency components.

[0110] In the process of performing first-level inverse wavelet reconstruction on the second-level low-frequency components and the second-level high-frequency components to obtain the enhanced and recovered low-frequency components, the following relationship exists:

[0111] ;

[0112] in, This indicates enhanced recovery of low-frequency components. This represents the convolution operation performed on the four components after wavelet decomposition. This indicates that the process is performed through the first-level inverse wavelet reconstruction.

[0113] In this process, the four components of the second-layer decomposition are convolved to extract features, and the first-layer inverse wavelet is used to reconstruct them to restore low-frequency features, so as to obtain the enhanced restored low-frequency components.

[0114] In the process of obtaining the output features of the wavelet convolution module through the second-layer inverse wavelet reconstruction of the low-frequency components, the first-layer high-frequency components, and the first-layer low-frequency components based on enhanced recovery, the following relationship exists:

[0115] ;

[0116] in, This represents the output features of the wavelet convolution module. This indicates that the process has undergone a second layer of inverse wavelet reconstruction.

[0117] In this process, the low-frequency features reconstructed by the second wavelet are... With the first layer of low-frequency features The features are fused and combined with the first-layer high-frequency features, and then reconstructed by the second-layer inverse wavelet to generate complete, multi-scale, and multi-directional enhanced style features;

[0118] In the process of performing a linear transformation on the initial style features to obtain the query vector of the matrix, the following relationship exists:

[0119] ;

[0120] in, A query vector representing a matrix;

[0121] In the process of stacking the initial style features and the enhanced style features and then performing a linear transformation to obtain the key vector and value vector of the matrix, the following relationship exists:

[0122] ;

[0123] in, Represents the key vector of a matrix. Represents the value vector of a matrix. This indicates that the two features will be merged along the newly added dimension. Indicates enhanced stylistic features;

[0124] In the process of obtaining attention weights by processing the query vector, key vector, and value vector of the matrix through an attention mechanism, the following relationship exists:

[0125] ;

[0126] in, and Both represent attention weights. This indicates that attention has been paid to the calculation;

[0127] In this process, based on the query vector of the matrix , matrix key vector And the value vector of the matrix Attention mechanisms are used to obtain the initial stylistic features of each spatial location. With enhanced style features The preference between the two types of features is used to dynamically adjust the fusion ratio of the two types of features;

[0128] In the process of using attention weights to perform weighted summation on the initial style features and the enhanced style features to obtain the weighted sum result, and then performing tensor dimension compression on the weighted sum result to obtain the style-enhanced style features, the following relationship exists:

[0129] ;

[0130] in, Stylistic features that indicate enhanced style. This indicates that the tensor dimension has been compressed.

[0131] Step 4: Input the style-enhanced style features into the Mamba-Transformer encoder module for processing to obtain deep style features;

[0132] In step 4, the style-enhanced style features are input into the Mamba-Transformer encoder module for processing to obtain deep style features. The specific steps are as follows:

[0133] The style-enhanced style features are input into the Mamba-Transformer encoder module and processed sequentially through three stacked Mamba-Transformer modules to obtain the global and local information of the style-enhanced style features.

[0134] The global and local information of the style features after style enhancement are processed sequentially through a feedforward network and layer normalization to obtain deep style features.

[0135] Step 5: Input the deep content features and deep style features into the decoder module for processing to obtain a face sketch image;

[0136] In step 5, the depth content features and depth style features are input into the decoder module for processing to obtain a facial sketch image, which specifically includes the following sub-steps:

[0137] The decoder module consists of three Transformer decoder modules and three convolutional decoder modules;

[0138] Each Transformer decoder module consists of three Transformer models;

[0139] Deep content features and deep style features are input into the first Transformer decoder module. The deep style features are used as the key and value vectors for attention calculation, and the deep content features are used as the query vector for attention calculation. The first Transformer model processes these features to obtain the first layer of Transformer output features. The deep style features are used as the key and value vectors for attention calculation, and the first layer of Transformer output features are used as the query vector for attention calculation. The second Transformer model processes these features to obtain the second layer of Transformer output features. The third Transformer output features are used as the first layer of fused features.

[0140] The first-layer fused features are input into the second Transformer decoder module for processing to obtain the second-layer fused features;

[0141] The second-layer fused features are then input into the third Transformer decoder module for processing to obtain the third-layer fused features, which are then used as the final fused features.

[0142] The final fused features are processed through three convolutional decoder modules to obtain a face sketch image; the workflow of the convolutional decoder module is 3×3 convolution operation and upsampling operation.

[0143] Please see Figure 5 This invention also provides a face sketching generation system based on Mamba and wavelet convolution, the system comprising:

[0144] Image processing module, used for:

[0145] The content image and style image are cropped separately to obtain cropped content images and style images; the cropped content images and style images are flattened and then processed by linear mapping to obtain preliminary content features and preliminary style features; position information is added to the preliminary content features to obtain preliminary content features with added position information.

[0146] The Mamba-Transformer encoder module is used for:

[0147] The initial content features after incorporating location information are processed to obtain deep content features;

[0148] The initial stylistic features are processed to obtain enhanced stylistic features;

[0149] Style-enhanced wavelet convolution module, used for:

[0150] The style features enhanced by style are processed to obtain deep style features;

[0151] Decoder module, used for:

[0152] Deep content features and deep style features are processed to obtain a facial sketch image.

[0153] For further details, please refer to Figure 4 This invention also conducted quantitative experiments on four datasets (CUHK, AR, XM2VTS, and CUGSF), using metrics such as Peak Signal-to-Noise Ratio (PSNR), Root Mean Square Error (RMSE), Learned Perceptual Patch Similarity (LPIPS), and Fréchet Inception Distance (FID) for evaluation. The quantitative experimental results are as follows: Figure 4 As shown, the best results are marked in bold, and the second-best results are marked with an underline. Figure 4 As can be seen, the proposed method achieved state-of-the-art results on all four datasets, fully demonstrating its excellent performance in detail preservation, visual similarity, and overall realism. The metrics results further validate the effectiveness of the proposed method.

[0154] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0155] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0156] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A face sketch generation method based on Mamba and wavelet convolution, characterized in that, The method is realized by a face sketch generation model, and the face sketch generation model is composed of an image processing module, a Mamba-Transformer encoder module, a style-enhanced wavelet convolution module, and a decoder module. Step 1, input the content image and the style image into the image processing module, and respectively crop the content image and the style image to obtain the cropped content image and the cropped style image; flatten the cropped content image and the cropped style image respectively and process by linear mapping to obtain the preliminary content feature and the preliminary style feature; The preliminary content feature is added with position information to obtain the preliminary content feature after adding position information; Step 2, input the preliminary content feature after adding position information into the Mamba-Transformer encoder module for processing to obtain the deep content feature; Step 3, input the preliminary style feature into the style-enhanced wavelet convolution module for processing to obtain the style-enhanced style feature; Step 4, input the style-enhanced style feature into the Mamba-Transformer encoder module for processing to obtain the deep style feature; Step 5, input the deep content feature and the deep style feature into the decoder module for processing to obtain the face sketch image; The specific steps are as follows: The Mamba-Transformer encoder module is composed of a visual Mamba module and a simplified variable size window attention module in parallel; The preliminary content feature after adding position information is processed by the state space model branch in the visual Mamba module to obtain the output feature after state space branch processing; The preliminary content feature after adding position information is processed by the symmetric convolution branch in the visual Mamba module to obtain the output feature after symmetric convolution branch processing; The output feature after state space branch processing and the output feature after symmetric convolution branch processing are fused to obtain the output feature after visual Mamba module processing; The preliminary content feature after adding position information is input into the simplified variable size window attention module for processing to obtain the predicted window size and the predicted window offset; The predicted window size, the predicted window offset, and the preliminary content feature after adding position information are used to obtain the dynamic window feature, the key matrix, and the value matrix; The dynamic window feature, the key matrix, and the value matrix are used to obtain the output feature after simplified variable size window attention module processing; The output feature after visual Mamba module processing and the output feature after simplified variable size window attention module processing are weighted and fused, and then processed by convolution to obtain the fusion feature; The fusion feature is sequentially processed by the feedforward neural network and the layer normalization to obtain the deep content feature.

2. The method for face sketch generation based on Mamba and wavelet convolution according to claim 1, characterized in that, In the step 1, the preliminary content feature is added with position information to obtain the preliminary content feature after adding position information, and the relationship formula corresponding to the process exists as follows: ； wherein, denotes the preliminary content features after adding the location information, denotes the preliminary content features, denotes the location encoding, denotes element-wise addition of the feature maps.

3. The method for face sketch generation based on Mamba and wavelet convolution according to claim 2, characterized in that, In the process of processing the preliminary content feature with added position information through the state space model branch in the visual Mamba module to obtain the output feature after state space branch processing, there are corresponding relationship formulas as follows: ； wherein, represents the output feature after state space branch processing, represents processing by a state space model, represents processing by a SiLU activation function, represents processing by a one-dimensional convolution layer, represents processing by a linear mapping; In the process of processing the preliminary content feature with added position information through the symmetric convolution branch in the visual Mamba module to obtain the output feature after symmetric convolution branch processing, there are corresponding relationship formulas as follows: ； wherein, denotes the output feature after processing by the symmetric convolution branch. In the process of fusing the output feature after state space branch processing and the output feature after symmetric convolution branch processing to obtain the output feature after processing by the visual Mamba module, there are corresponding relationship formulas as follows: ； wherein, represents the output feature after processing by the visual Mamba module, represents a concatenation process along the channel dimension; In the process of inputting the preliminary content feature with added position information into the simplified variable size window attention module for processing to obtain the predicted window size and the predicted window offset, there are corresponding relationship formulas as follows: ； wherein, represents a predicted window size, represents a predicted window offset, represents a convolution operation, represents a processing by an activation function with negative slope, represents a processing by global average pooling; In the process of obtaining the dynamic window feature, the key matrix and the value matrix by using the predicted window size, the predicted window offset and the preliminary content feature with added position information, there are corresponding relationship formulas as follows: ； wherein, denotes a dynamic window feature, denotes a spatial Transformer operation, denotes a key matrix, denotes a value matrix, denotes a reshape operation that flattens the feature map into a sequence; In the process of obtaining the output feature after processing by the simplified variable size window attention module by using the dynamic window feature, the key matrix and the value matrix, there are corresponding relationship formulas as follows: ； wherein, denotes a local key matrix obtained by window sampling, denotes a local value matrix obtained by window sampling, denotes a local attention computation, denotes an output feature after processing by the simplified variable-size window attention module, denotes processing by the linear window attention; In the process of weighting fusion of the output feature after processing by the visual Mamba module and the output feature after processing by the simplified variable size window attention module, and then processing by convolution to obtain the fusion feature, there are corresponding relationship formulas as follows: ； wherein, denotes a fused feature, and both denote trainable weights, denotes multiplication of corresponding position elements; In the process of sequentially performing feedforward neural network processing and layer normalization processing on the fusion feature to obtain the deep content feature, the following relationship formulas are included: ； wherein, represents a deep content feature, represents a layer normalization process, represents a feed-forward neural network process.

4. The method for face sketch generation based on Mamba and wavelet convolution according to claim 3, characterized in that, In step 3, the preliminary style feature is input into the style enhancement wavelet convolution module for processing to obtain the style-enhanced style feature, and the specific steps are as follows: The style enhancement wavelet convolution module is composed of a style enhancement convolution module and a wavelet convolution module in parallel; The preliminary style feature is processed by five kinds of difference convolution in the style enhancement wavelet convolution module for multi-directional feature extraction to obtain the output feature of the addition path and the output feature of the splicing path; The output features of the addition path and the splicing path are used to obtain the gating weight, and the output feature of the style enhancement convolution module is obtained by using the gating weight; The preliminary style feature is split by the wavelet convolution module to obtain the first layer low frequency component and the first layer high frequency component; The first layer low frequency component is decomposed by wavelet to obtain the second layer low frequency component and the second layer high frequency component; The first layer inverse wavelet reconstruction processing is performed on the second layer low frequency component and the second layer high frequency component to obtain the enhanced and recovered low frequency component; Based on the enhanced and recovered low frequency component, the first layer high frequency component and the first layer low frequency component, the second layer inverse wavelet reconstruction processing is performed to obtain the output feature of the wavelet convolution module; The output feature of the style enhancement convolution module and the output feature of the wavelet convolution module are subjected to channel attention processing and spatial attention weighted fusion processing to obtain the enhanced style feature; The preliminary style feature is linearly transformed to obtain the query vector of the matrix; stacking the preliminary style feature and the enhanced style feature and then performing linear transformation to obtain a matrix key vector and a matrix value vector; obtaining an attention weight through attention mechanism processing according to the matrix query vector, the matrix key vector and the matrix value vector; performing weighted summation on the preliminary style feature and the enhanced style feature by using the attention weight to obtain a weighted summation result, and then performing tensor dimension compression processing on the weighted summation result to obtain a style-enhanced style feature.

5. The method for face sketch generation based on Mamba and wavelet convolution according to claim 4, characterized in that, In the process of using the five difference convolutions in the style-enhanced wavelet convolution module to perform multi-directional feature extraction processing on the preliminary style feature to obtain the output feature of the addition path and the output feature of the concatenation path, the following relationship exists: ; wherein, represents an output feature of an addition path, represents a preliminary style feature, represents a convolution and an index, represents an output feature of a concatenation path, represents a second layer convolution processing, represents an output feature of a normal convolution, represents an output feature of a center difference convolution, represents an output feature of an angle difference convolution, represents an output feature of a horizontal difference convolution, represents an output feature of a vertical difference convolution; In the process of obtaining the gating weight by using the output feature of the addition path and the output feature of the concatenation path, and then obtaining the output feature of the style-enhanced convolution module by using the gating weight, the following relationship exists: ； wherein, denotes a gating weight, denotes an element-wise Sigmoid function processing, denotes the output feature of the style enhancement convolution module; In the process of splitting the preliminary style feature by using the wavelet convolution module to obtain a first layer low frequency component and a first layer high frequency component, the following relationship exists: ； wherein, represents the first layer low frequency component, , and all represent the first layer high frequency component, represents a convolution process through a corresponding wavelet filter. In the process of wavelet decomposing the first layer low frequency component to obtain a second layer low frequency component and a second layer high frequency component, the following relationship exists: ； wherein, represents the second layer low frequency component, , and all represent the second layer high frequency component, represents a second layer wavelet decomposition process; In the process of performing first layer inverse wavelet reconstruction processing on the second layer low frequency component and the second layer high frequency component to obtain an enhanced recovered low frequency component, the following relationship exists: ； wherein, denotes an enhanced recovery of low frequency components, denotes a convolution operation on the four components after wavelet decomposition, denotes a reconstruction by inverse wavelet of the first layer; In the process of obtaining the output feature of the wavelet convolution module through second layer inverse wavelet reconstruction processing based on the enhanced recovered low frequency component, the first layer high frequency component and the first layer low frequency component, the following relationship exists: ； wherein, represents the output feature of the wavelet convolution module, represents the second layer inverse wavelet reconstruction processing.

6. The method for face sketch generation based on Mamba and wavelet convolution according to claim 5, characterized in that, In the process of performing linear transformation on the preliminary style feature to obtain the matrix query vector, the following relationship exists: ； wherein query vector representing a matrix; In the process of stacking the preliminary style feature and the enhanced style feature and then performing linear transformation to obtain the matrix key vector and the matrix value vector, the following relationship exists: ； wherein, denotes a key vector of a matrix, denotes a value vector of a matrix, denotes merging of two features along a new dimension, denotes an enhanced style feature; In the process of obtaining the attention weight through attention mechanism processing according to the matrix query vector, the matrix key vector and the matrix value vector, the following relationship exists: ； wherein, and both represent attention weights, represents a passage through an attention computation; In the process of performing weighted summation on the preliminary style feature and the enhanced style feature by using the attention weight to obtain a weighted summation result, and then performing tensor dimension compression processing on the weighted summation result to obtain a style-enhanced style feature, the following relationship exists: ； wherein, denotes a style feature representing style augmentation, denotes a tensor dimension compression process.

7. The method for face sketch generation based on Mamba and wavelet convolution according to claim 6, characterized in that, In step 4, the style-enhanced style feature is input into the Mamba-Transformer encoder module for processing to obtain a deep style feature, and the specific steps are as follows: The style-enhanced style feature is input into the Mamba-Transformer encoder module and processed through three stacked Mamba-Transformer modules in sequence to obtain global information and local information of the style-enhanced style feature; The global information and the local information of the style-enhanced style feature are processed through a feedforward network and layer normalization in sequence to obtain a deep style feature.

8. The method for face sketch generation based on Mamba and wavelet convolution according to claim 7, characterized in that, In the step 5, the deep content feature and the deep style feature are input into the decoder module for processing to obtain the face sketch image, and the step 5 specifically comprises the following sub-steps: The decoder module is composed of three Transformer decoder modules and three convolutional decoder modules; Each Transformer decoder module is composed of three Transformer models; The deep content feature and the deep style feature are input into the first Transformer decoder module, the deep style feature is taken as the key vector and the value vector of attention calculation, the deep content feature is taken as the query vector of attention calculation, and the first layer Transformer output feature is obtained through the first Transformer model; The deep style feature is taken as the key vector and the value vector of attention calculation, and the first layer Transformer output feature is taken as the query vector of attention calculation, and the second layer Transformer output feature is obtained through the second Transformer model; The deep style feature is taken as the key vector and the value vector of attention calculation, and the second layer Transformer output feature is taken as the query of attention calculation, and the third layer Transformer output feature is obtained through the third Transformer model, and the third layer Transformer output feature is taken as the first layer fusion feature; The first layer fusion feature is input into the second Transformer decoder module for processing to obtain the second layer fusion feature; The second layer fusion feature is input into the third Transformer decoder module for processing to obtain the third layer fusion feature, and the third layer fusion feature is taken as the final fusion feature; The final fusion feature is processed through the three convolutional decoder modules to obtain the face sketch image; wherein the working process of the convolutional decoder module is 3*3 convolution operation and up-sampling operation.

9. A system for face sketch generation based on Mamba and wavelet convolution, characterized in that, The system applies the face sketch generation method based on Mamba and wavelet convolution of any one of claims 1 to 8, and the system comprises: An image processing module, configured to: crop the content image and the style image respectively to obtain the cropped content image and the cropped style image; flatten the cropped content image and the cropped style image respectively and process through linear mapping to obtain the preliminary content feature and the preliminary style feature; and add position information to the preliminary content feature to obtain the preliminary content feature with position information; A Mamba-Transformer encoder module, configured to: process the preliminary content feature with position information to obtain the deep content feature; process the preliminary style feature to obtain the style-enhanced style feature; A style-enhanced wavelet convolution module, configured to: process the style-enhanced style feature to obtain the deep style feature; A decoder module, configured to: process the deep content feature and the deep style feature to obtain the face sketch image.

Images