Hierarchical state space image style migration method and device based on Dunhuang style perception

Abstract

The invention provides a Mama image style migration method and device based on Dunhuang style perception. The method comprises the steps of S110 to S140: S110: collecting data of a Dunhuang mural style image and a natural content image, performing preprocessing, and constructing a training data set including the content image and the style image; s120, constructing a hierarchical state space network model suitable for Dunhuang mural style migration; s130, training the network model by using the training data set to obtain a trained model; and S140, inputting a to-be-processed content image and a target Dunhuang style image into the trained model, and dynamically modulating a time step parameter (dt) of the state space model by using a style vector to generate a stylized image fusing the Dunhuang mural style and the content structure. Therefore, the problems that when an existing style migration method is used for processing a high-resolution mural, video memory consumption is large, the long-range dependence capture capacity is weak, a specific color system of the Dunhuang mural is difficult to accurately restore, texture spectrum aliasing is likely to occur and the like are solved.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and graphics image processing technology, and more specifically, to a method and apparatus for hierarchical state space image style transfer based on Dunhuang style perception. Background Technology

[0002] With the rapid development of deep learning technology, neural style transfer has become a research hotspot in computer vision, with wide applications in artistic creation, entertainment, and digital preservation of cultural heritage. Gatys et al. pioneered the use of convolutional neural networks (VGG) to extract image features, achieving for the first time a neural network-based art style transfer, demonstrating that the correlation of deep features can express the artistic style of an image. Subsequently, Huang et al. proposed the Adaptive Instance Normalization (AdaIN) algorithm, which greatly improves the inference speed of arbitrary style transfer by aligning the mean and variance of content features and style features. Although these traditional convolutional neural network (CNN)-based methods have achieved significant results, they are limited by the local receptive field of the convolutional kernel, making it difficult to capture the long-range dependencies of images, resulting in a lack of coherence in the large-scale structure of the generated stylized images.

[0003] In recent years, Deng et al. proposed the Transformer-based style transfer network StyTr2 to address the shortcomings of CNNs in capturing long-range dependencies. This network introduces a self-attention mechanism, effectively improving the model's ability to capture global image features. However, the computational complexity of the self-attention mechanism is quadratic with image resolution, resulting in huge memory consumption and making it difficult to directly apply to high-resolution Dunhuang mural processing. Furthermore, existing style transfer algorithms often ignore the color and texture characteristics of specific art movements. (1) Color deviation: Dunhuang murals have a unique "red and green" color system (such as the unique hue after oxidation and color change). General style transfer often adopts global mean alignment, which can easily destroy the color semantics with historical accumulation.

[0004] (2) Texture aliasing: Existing texture loss calculations are mostly based on Gaussian pyramids, which can easily introduce spectral aliasing effects into the generated image, resulting in blurred lines or artifacts in the mural.

[0005] (3) Computational efficiency: When dealing with detailed Gigapixel-level images like the Dunhuang murals, existing models often struggle to maintain inference speed while also ensuring memory efficiency.

[0006] Therefore, there is an urgent need for a style transfer method that can both efficiently capture long-range contextual information and accurately protect the specific color and texture characteristics of Dunhuang murals. Summary of the Invention

[0007] The purpose of this disclosure is to provide a hierarchical state space image style transfer method based on Dunhuang style perception, which aims to solve the problems of high memory consumption, weak long-range dependency capture capability, difficulty in accurately restoring the unique color system of Dunhuang murals, and easy texture spectrum aliasing when processing high-resolution murals.

[0008] In general, a hierarchical state-space image style transfer method based on Dunhuang style perception is provided, including steps S110 to S140: S110: Collect style images and natural content images of Dunhuang murals, preprocess them, and construct a pairwise training dataset containing content images and style images; S120: Construct a hierarchical state-space network model suitable for style transfer of Dunhuang murals. The model includes a hierarchical encoder, a style embedder, and a multi-scale decoder. S130: The hierarchical state space network model is trained using the pairwise training dataset, and the model parameters are optimized by combining wavelet texture loss and edge enhancement HSV loss to obtain the trained model; S140: Input the content image to be processed and the target Dunhuang style image into the trained hierarchical state space network model, and use the style vector to dynamically modulate the time step parameters of the state space model to generate a stylized image that integrates the style and content structure of Dunhuang murals.

[0009] The method for constructing the hierarchical state-space network model is as follows: a hierarchical structure is used as the backbone network, and a visual state-space module (SS2D) is introduced into the building blocks. A multi-directional scanning mechanism is used to unfold the two-dimensional features into a one-dimensional sequence. A style-aware mechanism is introduced into the visual state-space module, and the style vector s extracted by the style embedder is used to set the discretized time step parameter of the state-space model. Dynamic modulation is performed; specifically, for input feature x, content-related time-step features are first generated using the input projection layer. Simultaneously, style-related time-step features are generated using the style vector s through linear mapping. The two values ​​are added together and then passed through the Softplus activation function to obtain the final effective time step. The calculation formula is: Using the modulated Discretizing the parameters of the continuous state space allows control over the dynamic propagation of feature information in the sequence, thereby enabling style-related long-range dependency modeling.

[0010] The multi-scale decoder also includes an HSV color space modulation module. This module first maps the input feature map to the HSV color space using a differentiable RGB-to-HSV transformation, and divides the feature channels into three groups corresponding to hue (H), saturation (S), and lightness (V). Using the modulation parameters generated by mapping the style vector through a fully connected layer, independent affine transformations are performed on the features of each of the three groups. For the hue group features… The target hue center and attraction intensity are predicted using style vectors, and nonlinear offset modulation is performed by calculating the annular distance between the pixel hue and the target hue; for saturation grouping features... and brightness grouping features The following linear modulation formula is used: Where s is the style vector. and These are the scaling factor and bias factor generated from the style vector, respectively.

[0011] The loss function L used in the training process consists of content loss. Style loss Wavelet texture loss and edge-enhanced HSV loss composition:

[0012] Among them, the wavelet texture loss The definition is as follows: using Discrete Wavelet Transform (DWT) to decompose the generated stylized image and the target stylized image into a low-frequency component LL and a high-frequency component set {LH, HL, HH}; discarding the low-frequency component LL which contains image structure, and only selecting the high-frequency component set which contains texture information; calculating the Gram matrix difference between the high-frequency components of the stylized image and the high-frequency components of the target stylized image.

[0013] The edge-enhanced HSV loss The definition is as follows: First, convert the generated image and the style image to the HSV color space, and separate the hue (H), saturation (S), and lightness (V) components; then, use the Sobel operator to extract the edge feature map of the style image. ; Calculate the differences between the generated image and the style image in each HSV component, and utilize edge feature maps Weighting is applied to the differences in hue and saturation.

[0014] In another general aspect, a hierarchical state-space image style transfer device based on Dunhuang style perception is provided, comprising: The image input and preprocessing module is used to acquire the content image to be processed and the target style image, and to perform size normalization and enhancement processing; The style transfer network processing module uses a hierarchical Mamba encoder to extract content features from the content image and style features from the style image, respectively. The stylization generation module includes a style-aware decoder that injects style encoding into the decoding process and performs style transfer by adjusting the SSM time step (dt) and HSV feature grouping. The results output and display module is used to output and display the generated stylized images.

[0015] The technical effects to be achieved by the embodiments of the present invention are as follows: This invention is of great significance for the style transfer and preservation of high-resolution art images such as the Dunhuang murals. Addressing the high memory consumption of existing Transformer models, this invention introduces a hierarchical state-space model (Mamba) with linear computational complexity. Through an innovative style-aware time step (dt) modulation mechanism, it achieves efficient long-range style dependency modeling at the underlying dynamics level. Furthermore, considering the unique color and texture system of the Dunhuang murals, this invention proposes HSV grouped color modulation and wavelet texture loss, effectively solving the color deviation and texture spectrum aliasing problems in traditional methods. This allows for the generation of stylized images with clear textures and accurate color reproduction while maintaining low memory consumption. Attached Figure Description

[0016] The above and other objects and features of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings.

[0017] Figure 1 This is a schematic diagram illustrating the architecture of a Dunhuang-style-aware Mamba image style transfer method according to an embodiment of the present disclosure; Figure 2 A detailed step-by-step schematic diagram of a Mamba image style transfer method based on Dunhuang style perception provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a Mamba image style transfer device based on Dunhuang style perception provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0019] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0020] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0021] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0022] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0023] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0024] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0025] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0026] Figure 1 This is a flowchart illustrating a style-aware Mamba-based method for style transfer of Dunhuang murals according to an embodiment of the present disclosure.

[0027] To achieve the aforementioned objectives, the present invention employs the following technical framework: Figure 1 As shown.

[0028] To facilitate understanding of this embodiment, a method for style transfer of Dunhuang murals based on style-aware Mamba, as disclosed in this embodiment of the invention, will first be described in detail. (See [link to relevant documentation]). Figure 1 The diagram shows a flowchart of a style transfer method for Dunhuang murals based on style-aware Mamba (see detailed flowchart). Figure 2 This method can be applied to an intelligent image style transfer system, which includes an image input and preprocessing module, a style transfer network processing module, and a result output and display module. The method mainly includes the following steps S110 to S140: S110: Collect Dunhuang mural style images and natural content images, and perform preprocessing (such as aging enhancement) to construct a pairwise training dataset containing content images and style images; S120: Construct a hierarchical state-space network model (Dh-SaMam) suitable for style transfer of Dunhuang murals. This model integrates a hierarchical Mamba encoder, a style embedder, and a multi-scale decoder that includes HSV color modulation. S130: Train the network model constructed in S120 using the paired training dataset, and optimize the model parameters by combining composite loss functions such as wavelet texture loss and edge enhancement HSV loss to obtain the trained weights. S140: Collect the content image to be processed and the target Dunhuang style image, and input them into the network model trained in S120. Using the weights obtained in S130, the state space model is dynamically modulated by the style vector to generate a stylized image that integrates the style and content structure of Dunhuang murals.

[0029] As a specific example, this application provides a method for style transfer of Dunhuang murals based on style-aware Mamba, see [link to relevant documentation]. Figure 2 This includes the following steps S21 to S26: S21: Data Collection and Preprocessing. Collect natural landscapes and figures as content images, and high-resolution digital images of the Dunhuang Mogao Grottoes murals as style images. Standardize the size of the collected images, uniformly adjusting them to 512×512 pixels.

[0030] In the training preprocessing stage, in addition to conventional random cropping and flipping, this invention designs a "Dunhuang Aging Augmentation" model specifically for the weathering and oxidation characteristics of the Dunhuang murals due to their age. This model defines a set of aging operators. During training, activation is performed randomly with a preset probability (e.g., P=0.3), specifically including: Oxidation fading operator ( This simulates the reduction in color saturation caused by pigment oxidation. The saturation component in the HSV color space is adjusted and multiplied by a random factor. : Weathering particle operator ( ): Simulates the grainy texture caused by the peeling of the ground layer in a mural. Additive Gaussian noise is superimposed on the normalized image tensor: Noise intensity .

[0031] Brown oxide operator ( This is a crucial step in simulating the unique lead oxide color change (from red to black / brown) and smoky effect characteristic of Dunhuang murals. A specific color matrix transformation is used to map the RGB space to a brownish-brown tone; the specific transformation matrix... The definition is as follows: To preserve some of the original color information, the final output uses a weighted fusion of the original image and the transformed image: Wherein the fusion coefficient .

[0032] S22: Construct a hierarchical Mamba encoder. In order to efficiently extract high-resolution image features, a hierarchical encoder based on the Visual State Space Model (VSSM) is constructed.

[0033] The encoder consists of three stages, each composed of a Patch Merging layer and several stacked VSSM modules. The input image is first mapped to a feature sequence through Patch Embedding, and then downsampled in each stage through Patch Merging (resolutions of H / 4, H / 8, and H / 16, respectively), while the number of channels is multiplied. Inside the VSSM module, a 2D selective scanning mechanism (SS2D) is used. This mechanism first maps the input features through a linear layer, flattens the 2D feature map into a 1D sequence using various scanning paths such as Zigzag, then performs context modeling using the state space equation formula (1), and finally restores the processed sequence to a 2D feature map. This hierarchical structure can capture long-distance dependencies of the image while maintaining linear computational complexity.

[0034] , Formula (1) describes the recursive reasoning process of the discretized linear state space model. The specific meanings of each variable are as follows: t: Represents the discrete time step index. In this embodiment, due to the use of the SS2D mechanism, t corresponds to the position index in the generated one-dimensional sequence after the two-dimensional image feature map is expanded by multi-directional (such as Zigzag) scanning; : Represents the input feature vector of the sequence at time t, corresponding to the local feature representation of the image at the current scanning position; : Represents the hidden state vector at time t. This vector is a high-dimensional latent variable used to store historical context information accumulated from the beginning of the sequence to the current time, and is the core carrier for the model to capture long-range dependencies; : Represents the hidden state vector of the previous time step t-1; : Represents the output feature vector at time t, that is, the feature representation after incorporating context information; A: Represents the discretized state transition parameter. It consists of a continuous state space matrix. Combined with time step It is obtained by discretization (such as the zero-order preservation method ZOH) and is used to control the evolution dynamics of the hidden state as the sequence progresses and the decay rate of historical information; B: Represents the Discrete Input Projection Parameter, used to project the current input features. Map and inject into the hidden state space; C: Represents the Output Projection Parameter, used to project the updated hidden state. Mapping the output feature space of the echo to obtain the final output .

[0035] S23: Construct a Style Embedder, which builds a separate style encoding branch to extract global style representations from the style image.

[0036] The style image is input into an encoder with a structure similar to S22 to extract the top-level feature map. Then, global average pooling is used to compress the spatial dimension, followed by mapping through a multilayer perceptron (MLP) to obtain a compact style latent vector. (Style Latent Vector). This vector contains high-level semantic information about the target mural, such as its hue, brushstrokes, and texture, which is used to guide the subsequent decoding process.

[0037] S24: Construct a Multi-Scale Style-Aware Decoder. This step constructs a U-Net architecture decoder with skip connections, aiming to integrate the multi-scale content features extracted in S22 with the global style vector generated in S23. Deep fusion is performed. The core component of the decoder is the "style-aware Mamba module," which is deployed at every scale level of the decoder. To overcome the content structure collapse and color aliasing problems caused by style transfer in existing technologies, this embodiment designs two key style injection mechanisms in the decoding module, the specific implementation of which is as follows: 1. Style-aware time step modulation mechanism (dt Modulation) In the Visual State Space Module (SS2D), the discretized time step parameter (dt) determines the propagation distance and forgetting rate of information in the sequence. Unlike traditional Mamba, which predicts based solely on the current input, (dt) is a more efficient and reliable predictor of information. This embodiment proposes a style-content dual-stream driven time step generation algorithm.

[0038] The algorithm utilizes the style vector generated by S23. The dynamic parameters of the state-space model are dynamically intervened, and the specific calculation process is recorded as follows: set up Let be the input feature sequence at time t. This is the global style vector. First, dynamic factors for content and style are generated using a parameter matrix. Map input features to content-related time factors At the same time, through the parameter matrix Map style vectors to style-related time factors : in, The operation broadcasts and copies the global style factor along the sequence length, making it consistent with... Dimension alignment.

[0039] The effective time step after style adjustment is then calculated, the two factors are fused, and the final time step is calculated using a modified Softplus activation function.

[0040] in, This is the scaling factor for the learnable style intensity. This is the bias term. Using the above formula, when the Dunhuang mural style vector is input, the model will adaptively adjust... The value. For example, for large areas of color in a mural, the model will generate a larger value. To capture long-range dependencies; while for dense line drawing areas, smaller values ​​are generated. To preserve high-frequency details.

[0041] 2. HSV Affine Modulation Based on Grouping Features To accurately reproduce the visual characteristics of Dunhuang murals—characterized by alternating red and green hues and outlines primarily relying on brightness—this embodiment abandons global feature modulation (such as AdaIN) during feature decoding and innovatively proposes a grouped HSV feature modulation strategy. This strategy does not mandate complete feature decoupling; instead, it divides feature channel C into three subgroups based on semantic priors. The response characteristics of hue, saturation, and value were simulated respectively, and different modulation formulas were applied to different groups.

[0042] First, style parameter mapping utilizes a multilayer perceptron (MLP) to map style vectors. The mapping is to three independent sets of affine transformation parameters (scaling factors). and bias factor ): Secondly, the strong style transfer of hue and saturation grouping affects the corresponding color information. and Grouping was performed using standard affine transformations to force the feature distribution to align with the color distribution of the Dunhuang murals: in This indicates element-wise multiplication. This step enables the transfer of the mural's unique colors (such as malachite green and ochre).

[0043] Finally, the structure-preserving gating for V-Group addresses the characteristic that the line drawing structure of Dunhuang murals is mainly reflected in the value channel. To prevent line loss during the stylization process, this embodiment uses... An adaptive residual gating formula was designed for the grouping.

[0044] First, calculate the fully stylized lightness features: Simultaneously, a structure-preserving gating coefficient g is predicted using style vectors: in This is the Sigmoid function. The final brightness output feature. It is formed by a weighted fusion of original features and stylized features: The gating value 'g' in this formula can sense the complexity of the style. When the target style is a complex gradient image, the model automatically increases the value of 'g', so that the output retains more of the original content's lightness structure (i.e., lines), thus effectively solving the persistent problem of "successful color transfer but blurred lines" in traditional methods.

[0045] S25: Construct a composite loss function.

[0046] To achieve end-to-end model training, this embodiment defines a multi-task composite loss function. Unlike general style transfer which only focuses on global statistics, this invention specifically designs frequency domain texture constraints and color space edge constraints to address the characteristics of Dunhuang murals. The total loss function is defined as follows: in , , , These are the weighting coefficients for each component of the loss. The specific design of each component loss is as follows: 1. Perceived content loss ( ) and style statistical loss ( The image features are extracted using a pre-trained VGG-19 network. The generated image is then calculated. With content images Euclidean distance over deep features (such as ReLU4_1) to preserve the main structure of the object: To leverage the streaming capabilities of the Mamba architecture, this embodiment employs a Mean-StdMatching strategy instead of computationally intensive Gram matrices. The generated image and style image are calculated. The mean across multiple feature layers (e.g., ReLU1_1 to ReLU4_1) ) and standard deviation ( )difference: This loss function forces the generated image to align with the global brushstroke distribution and color dynamic range of the Dunhuang murals.

[0047] 2. Wavelet texture loss ( To address the problem of blurred mural textures caused by Gaussian blurring in existing technologies, this embodiment proposes a frequency domain texture loss method based on Discrete Wavelet Transform (DWT). First, frequency domain decomposition is performed, using the Haar wavelet kernel to decompose image features into four frequency bands: Low-frequency component (LL): In the Dunhuang mural scene, this component corresponds to the overall outline, dharma structure, and composition of the Buddha image. To prevent stylization from destroying the image content, this loss function explicitly discards this component and does not impose stylistic constraints on it.

[0048] High-frequency component set ({LH, HL, HH}): This set corresponds to the high-frequency texture details of the Dunhuang murals. Specifically, LH captures horizontal brushstrokes, HL captures vertical lines, and HH captures diagonal pigment grains and cracks.

[0049] Secondly, high-frequency Gram matrix matching only calculates Gram matrix differences for high-frequency components to force the model to learn the microscopic texture of the mural:

[0050] Explanation of the meaning of formula symbols: k: Frequency band index, traversing the three high-frequency sub-bands: horizontal, vertical, and diagonal; The coefficient map of the generated image in the k-th frequency band after wavelet decomposition represents the texture intensity of the generated image in that frequency band. The feature coefficient map of the target Dunhuang-style image in the k-th frequency band serves as the benchmark for texture learning (Ground Truth). Gram matrix computation operator, which calculates the correlation matrix between feature channels, used to capture the statistical patterns of textures rather than pixel locations; Frobenius norm.

[0051] 3. Edge-enhanced HSV loss ( To address the color aliasing caused by RGB space calculation loss (such as confusing vermilion red in a mural with oxidized lead black), this embodiment incorporates edge-weighted loss in the HSV space.

[0052] First, the Sobel operator is used to extract the edge intensity map of the style image. And perform normalization: in For edge enhancement coefficients (e.g.) =5), which makes the weight of the edge region significantly higher than that of the flat region.

[0053] Next, the weighted color difference is calculated by taking the weighted mean square error of the generated image and the style image across the three HSV channels: Where p is the pixel index. This formula is derived from... By assigning higher loss weights to the mural line areas, it is ensured that while transferring colors, not only are the colors accurate (H / S component fitting), but also the generated mural lines are clear due to the strong constraint of the lightness (V component) on the edge weights.

[0054] S26: Model Training and Inference. The model is trained using the constructed pairwise datasets. The AdamW optimizer is used, with an initial learning rate set to... It also adjusts the learning rate in conjunction with a cosine annealing strategy.

[0055] During training, gradient accumulation is employed to accommodate larger batch sizes under memory constraints, and exponential moving average (EMA) is used to update model weights to improve generalization ability and generation quality. During inference, arbitrary content images and the target Dunhuang mural image are input into the trained network, and a high-resolution, stylistically realistic art image is generated after a single forward propagation.

[0056] In summary, this invention effectively solves the problems of long-range dependency capture difficulty, inaccurate color reproduction, and loss of texture details in the style migration of Dunhuang murals by combining the efficient sequence modeling capabilities of the hierarchical Mamba architecture with the explicit modulation of the HSV color space.

[0057] This invention also provides a style transfer device for Dunhuang murals based on style-aware Mamba, see [link / reference]. Figure 3 The device includes: Image input and preprocessing module 310 is used to acquire the content image and style image to be processed, and to perform size normalization and enhancement processing; The hierarchical feature encoding module 320 is used to extract the content features of the content image and the style features of the style image respectively using the hierarchical Mamba encoder; The stylization generation module 330 includes a style-aware decoder, which injects style encoding into the decoding process and performs style transfer by adjusting the SSM time step and HSV feature groups. The result output display module 340 is used to output the generated high-resolution Dunhuang-style image.

[0058] The embodiments of this application provide specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiments can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. The style-aware Mamba-based Dunhuang mural style transfer device provided in this application has the same technical features as the Dunhuang style-aware Mamba image style transfer method provided in the foregoing embodiments, and therefore can solve the same technical problems and achieve the same technical effects.

[0059] This application also provides an electronic device, specifically, the electronic device includes a processor and a storage device; the storage device stores a computer program, and the computer program, when run by the processor, executes the method described in any of the above embodiments.

[0060] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 400 includes: a processor 40, a memory 41, a bus 42, and a communication interface 43. The processor 40, the communication interface 43, and the memory 41 are connected through the bus 42. The processor 40 is used to execute executable modules, such as computer programs, stored in the memory 41.

[0061] The memory 41 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 43 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0062] Bus 42 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0063] The memory 41 is used to store programs. After receiving an execution instruction, the processor 40 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0064] Processor 40 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 40 or by instructions in software form. Processor 40 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 41. The processor 40 reads the information in memory 41 and, in conjunction with its hardware, completes the steps of the above method.

[0065] Corresponding to the above method, this application embodiment also provides a computer-readable storage medium storing machine-executable instructions. When the machine-executable instructions are called and run by a processor, the machine-executable instructions cause the processor to perform the steps of the above method.

[0066] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0067] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0068] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0069] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0070] It should be noted that similar reference numerals and letters in the accompanying drawings indicate similar items. Therefore, once an item is defined in one accompanying drawing, it does not need to be further defined and explained in subsequent accompanying drawings. In addition, the terms "first," "second," "third," etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0071] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A hierarchical state-space image style transfer method based on Dunhuang style perception, characterized in that, Including steps S110 to S140: S110: Collect style images and natural content images of Dunhuang murals, preprocess them, and construct a pairwise training dataset containing content images and style images; S120: Construct a hierarchical state space network model suitable for style transfer of Dunhuang murals. The model includes a hierarchical encoder based on the visual state space model, a style embedder, and a multi-scale decoder with jump connections. S130: The hierarchical state space network model is trained using the pairwise training dataset, and the model parameters are optimized by combining wavelet texture loss and edge enhancement HSV loss to obtain the trained model; S140: Input the content image to be processed and the target Dunhuang style image into the trained hierarchical state space network model, and use the style vector to dynamically modulate the time step parameters of the state space model to generate a stylized image that integrates the style and content structure of Dunhuang murals.

2. The hierarchical state-space image style transfer method based on Dunhuang style perception as described in claim 1, characterized in that, The process of collecting Dunhuang mural style images and natural content image data and performing preprocessing is as follows: high-definition digital images of Dunhuang Mogao Grottoes murals are collected as a style image set, and the collected images are standardized in size and uniformly adjusted to 512×512 pixels. In the training preprocessing stage, a Dunhuang aging enhancement model was designed, and a set of aging operators were defined. During training, activation is performed randomly with a preset probability, specifically including: Oxidation fading operator This simulates the reduction in color saturation caused by pigment oxidation by adjusting the saturation component in the HSV color space and multiplying it by a random factor. : Weathering Particle Operator To simulate the grainy texture caused by the peeling of the ground layer in a mural, additive Gaussian noise is added to the normalized image tensor. Noise intensity ; Brown oxide operator The key steps in simulating the unique lead oxide color change and smoky effect of Dunhuang murals involve using a specific color matrix transformation to map the RGB space to a brownish tone. The specific transformation matrix... The definition is as follows: To preserve some of the original color information, the final output uses a weighted fusion of the original image and the transformed image: Wherein the fusion coefficient .

3. The hierarchical state-space image style transfer method based on Dunhuang style perception as described in claim 1, characterized in that, The hierarchical state-space network model is constructed as follows: a hierarchical structure is used as the backbone network, and the hierarchical encoder contains multiple stages. Each stage reduces feature resolution and increases channel dimension through a downsampling layer. A visual state-space module is introduced into the building blocks of the encoder and decoder. The visual state-space module uses a multi-directional scanning mechanism to unfold two-dimensional image features into a one-dimensional sequence for sequence modeling. A style-aware mechanism is introduced into the visual state-space module, using style vectors extracted by a style embedder. Discretization time step parameters of the state-space model Dynamic modulation is performed; specifically, for the input features First, the input projection layer is used to generate content-related time step features. At the same time, style vectors are used Style-related time-step features are generated through linear mapping. The two values ​​are added together and then passed through the Softplus activation function to obtain the final effective time step. The calculation formula is: Using the modulated Discretizing the parameters of the continuous state space allows control over the dynamic propagation of feature information in the sequence, thereby enabling style-related long-range dependency modeling.

4. The hierarchical state-space image style transfer method based on Dunhuang style perception as described in claim 1, characterized in that, The multi-scale decoder incorporates two key style injection mechanisms in its decoding module, with the specific implementation details as follows: (1) Style-aware temporal step modulation mechanism: A style-content dual-stream driven temporal step generation algorithm is proposed, which utilizes the style vector The dynamic parameters of the state-space model are dynamically intervened, and the specific calculation process is recorded as follows: set up Let be the input feature sequence at time t. For the global style vector, firstly, dynamic factors of content and style are generated through a parameter matrix. Map input features to content-related time factors At the same time, through the parameter matrix Map style vectors to style-related time factors : in, The operation broadcasts and copies the global style factor along the sequence length, making it consistent with... Dimension alignment; The effective time step after style adjustment is then calculated, the two factors are fused, and the final time step is calculated using a modified Softplus activation function. : in, This is the scaling factor for the learnable style intensity. For bias terms; (2) HSV affine modulation mechanism based on grouped features: In the feature decoding process, a grouped HSV feature modulation strategy is proposed, which divides the feature channel C into three subgroups based on semantic prior. The response characteristics of hue, saturation, and brightness were simulated respectively, and different modulation formulas were applied to different groups. The first step is style parameter mapping, which uses a multilayer perceptron to map style vectors. The mapping is represented by three independent sets of affine transformation parameters, where Scaling factor Bias factor: ， ， Secondly, there is strong style transfer through hue and saturation grouping, for the corresponding color information. and Grouping was performed using standard affine transformations to force the feature distribution to align with the color distribution of the Dunhuang murals: in This indicates element-wise multiplication; Finally, there is the structural residual gating for brightness grouping. Design an adaptive residual gating formula in the grouping: First, calculate the fully stylized lightness features: Simultaneously, a structure-preserving gating coefficient g is predicted using style vectors: in For the Sigmoid function; Final brightness output features It is formed by a weighted fusion of original features and stylized features: When the target style is a complex line-based graph, the size will automatically increase. value.

5. The hierarchical state-space image style transfer method based on Dunhuang style perception as described in claim 1, characterized in that, The loss function in step S130 is implemented by constructing a composite loss function, defining a multi-task composite loss function. To address the unique characteristics of the Dunhuang murals, frequency domain texture constraints and color space edge constraints were specifically designed. The total loss function is defined as follows: in , , , These are the weighting coefficients for each component of the loss. The specific design of each component of the loss is as follows: Perceived content loss Style statistical loss Image features are extracted using a pre-trained VGG-19 network, and the generated image is calculated. With content images Euclidean distance over deep features to preserve the main structure of the object: The generated image and style image are calculated using a feature statistics matching strategy. Mean across multiple feature layers and standard deviation difference: This loss function forces the generated image to align with the global brushstroke distribution and color dynamic range of the Dunhuang murals; Wavelet texture loss This paper proposes a frequency domain texture loss based on discrete wavelet transform. First, frequency domain decomposition is performed. The image features are decomposed into four frequency bands using the Haar wavelet kernel: low frequency component LL and high frequency component set {LH, HL, HH}. LH captures horizontal brush strokes, HL captures vertical lines, and HH captures the pigment graininess and crack marks in the diagonal direction. Secondly, high-frequency Gram matrix matching only calculates Gram matrix differences for high-frequency components to force the model to learn the microscopic texture of the mural: Explanation of the meaning of formula symbols: k: Frequency band index, traversing the three high-frequency sub-bands: horizontal, vertical, and diagonal; The feature coefficient map of the generated image in the k-th frequency band after wavelet decomposition represents the texture intensity of the generated image in that frequency band. The feature coefficient map of the target Dunhuang-style image in the k-th frequency band serves as the benchmark for texture learning. Gram matrix computation operator, which calculates the correlation matrix between feature channels, is used to capture the statistical patterns of textures rather than pixel locations; :Frobenius norm; Edge-enhanced HSV loss Edge-weighted loss is designed in HSV space: First, the Sobel operator is used to extract the edge intensity map of the style image. And perform normalization: in For edge enhancement coefficients (e.g.) =5), which makes the weight of the edge region significantly higher than that of the flat region; Next, the weighted color difference is calculated by taking the weighted mean square error of the generated image and the style image across the three HSV channels: Where p is the pixel index.

6. A hierarchical state-space image style transfer device based on Dunhuang style perception, characterized in that, The hierarchical state-space image style transfer method based on Dunhuang style perception, as described in any one of claims 1-5, comprises: The image input and preprocessing module is used to acquire the content image to be processed and the target style image, and to perform size normalization and enhancement processing; The style transfer network processing module uses a hierarchical Mamba encoder to extract content features from the content image and style features from the style image, respectively. The stylization generation module includes a style-aware decoder that injects style encoding into the decoding process and performs style transfer by adjusting the SSM time step dt and HSV feature grouping. The results output and display module is used to output and display the generated stylized images.

Images